[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:56:40 dist-test-slave-30s6 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:56:40 dist-test-slave-30s6 krb5kdc[399](info): set up 2 sockets
Sep 29 00:56:40 dist-test-slave-30s6 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 00:56:41.211766 392 mini_kdc.cc:121] Time spent starting KDC: real 2.673s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 00:56:41.343111 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.131s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 00:56:41.560190 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.217s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 00:56:41.785214 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.225s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/test-user.keytab.
W20260929 00:56:42.499814 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.714s user 0.000s sys 0.002s
Sep 29 00:56:42 dist-test-slave-30s6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:56:42.545426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44867
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:42.652119 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:42.652396 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:42.652436 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:42.656533 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:42.656585 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:42.656605 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:42.656639 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:42.656672 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:42.661854 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:42.662842 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:42.663865 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:56:42.672780 412 server_base.cc:1034] running on GCE node
W20260929 00:56:42.672906 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:42.672957 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:42.672986 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:42.673682 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:42.674832 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:42.675981 412 hybrid_clock.cc:648] HybridClock initialized: now 1790643402675919 us; error 78 us; skew 500 ppm
Sep 29 00:56:42 dist-test-slave-30s6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:42.678805 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:56:42.679713 412 webserver.cc:460] Webserver started at http://127.0.98.62:45199/ using document root <none> and password file <none>
I20260929 00:56:42.680250 412 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:42.680310 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:42.680546 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:42.682574 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "168b93b6f3f8434f96262a692c970195"
format_stamp: "Formatted at 2026-09-29 00:56:42 on dist-test-slave-30s6"
server_key: "08e98dd02e567aaf7f1eb4bc98000605"
server_key_iv: "1379fbf0539d3d4e47a24be131122e8c"
server_key_version: "encryptionkey@0"
I20260929 00:56:42.683128 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "168b93b6f3f8434f96262a692c970195"
format_stamp: "Formatted at 2026-09-29 00:56:42 on dist-test-slave-30s6"
server_key: "08e98dd02e567aaf7f1eb4bc98000605"
server_key_iv: "1379fbf0539d3d4e47a24be131122e8c"
server_key_version: "encryptionkey@0"
I20260929 00:56:42.686647 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:42.688859 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:42.689978 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:42.690094 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "168b93b6f3f8434f96262a692c970195"
format_stamp: "Formatted at 2026-09-29 00:56:42 on dist-test-slave-30s6"
server_key: "08e98dd02e567aaf7f1eb4bc98000605"
server_key_iv: "1379fbf0539d3d4e47a24be131122e8c"
server_key_version: "encryptionkey@0"
I20260929 00:56:42.690236 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:42.716952 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:42.720393 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:42.720598 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:42.727407 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44867
I20260929 00:56:42.727427 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44867 every 8 connection(s)
I20260929 00:56:42.728351 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:42.731165 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:42.731684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 412
I20260929 00:56:42.731846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:56:42.733448 392 external_mini_cluster.cc:1428] Setting key 22c3a7fa047c508555349e96b22a2c2f
I20260929 00:56:42.737826 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195: Bootstrap starting.
I20260929 00:56:42.740056 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:42.740767 480 log.cc:824] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:42.742549 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195: No bootstrap required, opened a new log
I20260929 00:56:42.745290 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168b93b6f3f8434f96262a692c970195" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } }
I20260929 00:56:42.745528 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:42.745601 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 168b93b6f3f8434f96262a692c970195, State: Initialized, Role: FOLLOWER
I20260929 00:56:42.745999 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168b93b6f3f8434f96262a692c970195" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } }
I20260929 00:56:42.746140 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:56:42.746248 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:56:42.746357 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:42.747186 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168b93b6f3f8434f96262a692c970195" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } }
I20260929 00:56:42.747517 480 leader_election.cc:302] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 168b93b6f3f8434f96262a692c970195; no voters:
I20260929 00:56:42.747788 480 leader_election.cc:288] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:56:42.747903 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:42.748138 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [term 1 LEADER]: Becoming Leader. State: Replica: 168b93b6f3f8434f96262a692c970195, State: Running, Role: LEADER
I20260929 00:56:42.748497 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168b93b6f3f8434f96262a692c970195" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } }
I20260929 00:56:42.748927 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:42.750198 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 168b93b6f3f8434f96262a692c970195. Latest consensus state: current_term: 1 leader_uuid: "168b93b6f3f8434f96262a692c970195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168b93b6f3f8434f96262a692c970195" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } } }
I20260929 00:56:42.750335 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:42.750371 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "168b93b6f3f8434f96262a692c970195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168b93b6f3f8434f96262a692c970195" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } } }
I20260929 00:56:42.750470 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:42.751286 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 00:56:42 dist-test-slave-30s6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:42.753990 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:42.758987 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:42.737214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.0.98.62:44867)
0929 00:56:42.737697 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:42.737720 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:42.738729 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 00:56:42.741776 (+ 3047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:42.741790 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:42.742075 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:42.742428 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:42.744442 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:42.745863 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:42.747198 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:42.747808 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:42.747972 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:42.753154 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:42.753161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:42.753210 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:42.753241 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:42.755357 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:42.756005 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:42.756010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:42.756015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:42.756153 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:42.756561 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:42.756565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:42.756567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:42.757048 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:42.757267 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:42.757853 (+ 586us) server_negotiation.cc:299] Negotiation successful
0929 00:56:42.758115 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":137,"threads_started":1}
I20260929 00:56:42.759723 494 catalog_manager.cc:1347] Generated new cluster ID: dc319ef23d45403b96e73698e3a7ac8e
I20260929 00:56:42.759793 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:42.777048 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:42.778038 494 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:42.786568 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 168b93b6f3f8434f96262a692c970195: Generated new TSK 0
I20260929 00:56:42.787405 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:56:42.816537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:42.924798 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:42.925037 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:42.925079 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:42.929010 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:42.929068 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:42.929124 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:56:42.934149 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:42.935144 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:42.936110 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:42.945483 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:42.945488 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:42.945488 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:42.945988 506 server_base.cc:1034] running on GCE node
I20260929 00:56:42.946413 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:42.947021 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:42.948173 506 hybrid_clock.cc:648] HybridClock initialized: now 1790643402948154 us; error 28 us; skew 500 ppm
Sep 29 00:56:42 dist-test-slave-30s6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:42.950991 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:56:42.951830 506 webserver.cc:460] Webserver started at http://127.0.98.1:40645/ using document root <none> and password file <none>
I20260929 00:56:42.952363 506 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:42.952446 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:42.952653 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:42.954655 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "a8ea84e17a994ed8b34881039ddd4306"
format_stamp: "Formatted at 2026-09-29 00:56:42 on dist-test-slave-30s6"
server_key: "08faf0ca8204587b1cac1f8cf6dac801"
server_key_iv: "8750d03dbfb98b70efc2c53324892694"
server_key_version: "encryptionkey@0"
I20260929 00:56:42.955214 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a8ea84e17a994ed8b34881039ddd4306"
format_stamp: "Formatted at 2026-09-29 00:56:42 on dist-test-slave-30s6"
server_key: "08faf0ca8204587b1cac1f8cf6dac801"
server_key_iv: "8750d03dbfb98b70efc2c53324892694"
server_key_version: "encryptionkey@0"
I20260929 00:56:42.958865 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:56:42.961097 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:42.962246 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:56:42.962369 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "a8ea84e17a994ed8b34881039ddd4306"
format_stamp: "Formatted at 2026-09-29 00:56:42 on dist-test-slave-30s6"
server_key: "08faf0ca8204587b1cac1f8cf6dac801"
server_key_iv: "8750d03dbfb98b70efc2c53324892694"
server_key_version: "encryptionkey@0"
I20260929 00:56:42.962468 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:42.971928 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:42.974578 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:42.974720 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:42.975176 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:42.975910 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:42.975953 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:42.975988 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:42.976011 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:42.984535 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35461
I20260929 00:56:42.984563 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35461 every 8 connection(s)
I20260929 00:56:42.985397 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:56:42.991895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 506
I20260929 00:56:42.992012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:56:42.992265 392 external_mini_cluster.cc:1428] Setting key 22d0dae0a82e7251368635a6dcf0e22b
Sep 29 00:56:42 dist-test-slave-30s6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:42.998900 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:42.987004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32771 (local address 127.0.98.62:44867)
0929 00:56:42.987121 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:42.987124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:42.987843 (+ 719us) server_negotiation.cc:407] Connection header received
0929 00:56:42.989070 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:42.989073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:42.989118 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:42.989191 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:42.990244 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:42.990758 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:42.991828 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:42.992133 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:42.992255 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:42.994724 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:42.994727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:42.994730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:42.994755 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:42.996418 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:42.996994 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:42.997000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:42.997005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:42.997109 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:42.997495 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:42.997501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:42.997506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:42.997795 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:42.997902 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:42.998457 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 00:56:42.998619 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 00:56:42.999476 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:42.987301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44867 (local address 127.0.98.1:32771)
0929 00:56:42.987701 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:42.987739 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:42.988890 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:42.989323 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:42.989329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:42.989648 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:42.990093 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:42.990105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:42.990858 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:42.990860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:42.991704 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:42.991713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:42.992287 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:42.992291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:42.992455 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:42.993082 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:42.993097 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:42.994547 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:42.996532 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:42.996538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:42.996552 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:42.996867 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:42.997210 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:42.997213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:42.997215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:42.997376 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:42.997914 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:42.997919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:42.998138 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 00:56:42.998367 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 00:56:42.998614 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":109,"threads_started":1}
I20260929 00:56:43.001048 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44867
I20260929 00:56:43.001309 635 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:43.001780 635 heartbeater.cc:511] Master 127.0.98.62:44867 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:56:43.003379 444 ts_manager.cc:194] Registered new tserver with Master: a8ea84e17a994ed8b34881039ddd4306 (127.0.98.1:35461)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 00:56:43.004709 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:56:43.021147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:43.132948 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:43.133232 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:43.133284 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:43.137588 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:43.137655 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:43.137717 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:56:43.143410 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:43.144430 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:43.145427 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:43.155352 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:43.155395 640 server_base.cc:1034] running on GCE node
W20260929 00:56:43.155356 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:43.155356 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:43.155995 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:43.156672 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:43.157829 640 hybrid_clock.cc:648] HybridClock initialized: now 1790643403157797 us; error 45 us; skew 500 ppm
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:43.160636 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:56:43.161507 640 webserver.cc:460] Webserver started at http://127.0.98.2:45305/ using document root <none> and password file <none>
I20260929 00:56:43.162225 640 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:43.162334 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:43.162549 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:43.164592 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "929f0fe755ca40f7bf0274b7ce4c15fe"
format_stamp: "Formatted at 2026-09-29 00:56:43 on dist-test-slave-30s6"
server_key: "878c7c0f5cfb3cc19e90e57f4bc8a16e"
server_key_iv: "6046a3570bde951f36991e9fce6f86ed"
server_key_version: "encryptionkey@0"
I20260929 00:56:43.165220 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "929f0fe755ca40f7bf0274b7ce4c15fe"
format_stamp: "Formatted at 2026-09-29 00:56:43 on dist-test-slave-30s6"
server_key: "878c7c0f5cfb3cc19e90e57f4bc8a16e"
server_key_iv: "6046a3570bde951f36991e9fce6f86ed"
server_key_version: "encryptionkey@0"
I20260929 00:56:43.169169 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 00:56:43.171545 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:43.172787 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:56:43.172881 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "929f0fe755ca40f7bf0274b7ce4c15fe"
format_stamp: "Formatted at 2026-09-29 00:56:43 on dist-test-slave-30s6"
server_key: "878c7c0f5cfb3cc19e90e57f4bc8a16e"
server_key_iv: "6046a3570bde951f36991e9fce6f86ed"
server_key_version: "encryptionkey@0"
I20260929 00:56:43.172976 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:43.188385 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:43.191946 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:43.192085 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:43.192659 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:43.193461 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:43.193506 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:43.193544 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:43.193594 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:43.203269 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34487
I20260929 00:56:43.203299 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34487 every 8 connection(s)
I20260929 00:56:43.204116 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:56:43.207203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 640
I20260929 00:56:43.207316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:56:43.207592 392 external_mini_cluster.cc:1428] Setting key ada6562576d116ebb4bacf5561e28b44
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:56:43.218482 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:43.206108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51241 (local address 127.0.98.62:44867)
0929 00:56:43.206290 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:43.206294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:43.207092 (+ 798us) server_negotiation.cc:407] Connection header received
0929 00:56:43.208629 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:43.208632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:43.208682 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:43.208762 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:43.209907 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.210506 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.211658 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.212067 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.212189 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:43.214752 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:43.214759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:43.214765 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:43.214797 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:43.216466 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:43.216946 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:43.216951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:43.216956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:43.217054 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:43.217385 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:43.217390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:43.217392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:43.217627 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:43.217729 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:43.218171 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 00:56:43.218329 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 00:56:43.219367 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:43.206517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44867 (local address 127.0.98.2:51241)
0929 00:56:43.206936 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:43.206993 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:43.208398 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:43.208942 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:43.208949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:43.209310 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:43.209771 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:43.209783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.210627 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.210635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:43.211533 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:43.211542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.212208 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.212211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:43.212351 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:43.212921 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:43.212936 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:43.214583 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:43.216577 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:43.216582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:43.216592 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:43.216840 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:43.217150 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:43.217152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:43.217154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:43.217280 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:43.217738 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:43.217743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:43.217952 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 00:56:43.218118 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 00:56:43.218441 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:56:43.220593 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44867
I20260929 00:56:43.220835 769 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:43.221295 769 heartbeater.cc:511] Master 127.0.98.62:44867 requested a full tablet report, sending...
I20260929 00:56:43.222427 444 ts_manager.cc:194] Registered new tserver with Master: 929f0fe755ca40f7bf0274b7ce4c15fe (127.0.98.2:34487)
I20260929 00:56:43.222993 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:56:43.235451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:43.347819 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:43.348101 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:43.348182 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:43.352499 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:43.352571 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:43.352674 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:56:43.357865 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:43.358927 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:43.359920 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:43.369246 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:43.369270 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:43.369320 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:43.369467 774 server_base.cc:1034] running on GCE node
I20260929 00:56:43.369861 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:43.370528 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:43.371685 774 hybrid_clock.cc:648] HybridClock initialized: now 1790643403371673 us; error 24 us; skew 500 ppm
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:43.374382 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:56:43.375249 774 webserver.cc:460] Webserver started at http://127.0.98.3:37049/ using document root <none> and password file <none>
I20260929 00:56:43.375828 774 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:43.375927 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:43.376159 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:43.378109 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "42586ffd1e0e4e538b81c64fa3f6764e"
format_stamp: "Formatted at 2026-09-29 00:56:43 on dist-test-slave-30s6"
server_key: "04665e57f358e7ae4b50ab4c79c17f88"
server_key_iv: "0120c7173559187803e40286ceecbccc"
server_key_version: "encryptionkey@0"
I20260929 00:56:43.378741 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "42586ffd1e0e4e538b81c64fa3f6764e"
format_stamp: "Formatted at 2026-09-29 00:56:43 on dist-test-slave-30s6"
server_key: "04665e57f358e7ae4b50ab4c79c17f88"
server_key_iv: "0120c7173559187803e40286ceecbccc"
server_key_version: "encryptionkey@0"
I20260929 00:56:43.383389 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:56:43.385658 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:43.386858 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:56:43.386981 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "42586ffd1e0e4e538b81c64fa3f6764e"
format_stamp: "Formatted at 2026-09-29 00:56:43 on dist-test-slave-30s6"
server_key: "04665e57f358e7ae4b50ab4c79c17f88"
server_key_iv: "0120c7173559187803e40286ceecbccc"
server_key_version: "encryptionkey@0"
I20260929 00:56:43.387085 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:43.409760 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:43.412498 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:43.412700 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:43.413281 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:43.414072 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:43.414155 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:43.414255 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:43.414304 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:43.423645 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44329
I20260929 00:56:43.423655 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44329 every 8 connection(s)
I20260929 00:56:43.424510 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:56:43.431378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 774
I20260929 00:56:43.431497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:56:43.431757 392 external_mini_cluster.cc:1428] Setting key 2e4c747dd972cd84617a816653eb55a2
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:43.437573 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:43.426228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36929 (local address 127.0.98.62:44867)
0929 00:56:43.426363 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:43.426366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:43.427169 (+ 803us) server_negotiation.cc:407] Connection header received
0929 00:56:43.428503 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:43.428506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:43.428563 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:43.428667 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:43.429788 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.430374 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.431489 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.431760 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.431880 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:43.434644 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:43.434647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:43.434649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:43.434689 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:43.435854 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:43.436283 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:43.436285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:43.436286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:43.436355 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:43.436645 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:43.436647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:43.436649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:43.436807 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:43.436879 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:43.437307 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 00:56:43.437415 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:56:43.438434 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:43.426604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44867 (local address 127.0.98.3:36929)
0929 00:56:43.427007 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:43.427060 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:43.428290 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:43.428833 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:43.428840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:43.429198 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:43.429653 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:43.429666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.430489 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.430492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:43.431357 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:43.431365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.431939 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.431944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:43.432136 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:43.432817 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:43.432839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:43.434504 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:43.435944 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:43.435948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:43.435959 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:43.436191 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:43.436445 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:43.436447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:43.436448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:43.436567 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:43.436887 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:43.436891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:43.437097 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 00:56:43.437251 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 00:56:43.437484 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
I20260929 00:56:43.439405 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44867
I20260929 00:56:43.439620 903 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:43.440068 903 heartbeater.cc:511] Master 127.0.98.62:44867 requested a full tablet report, sending...
I20260929 00:56:43.440968 444 ts_manager.cc:194] Registered new tserver with Master: 42586ffd1e0e4e538b81c64fa3f6764e (127.0.98.3:44329)
I20260929 00:56:43.441526 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36929
I20260929 00:56:43.446033 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:56:43.471868 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:43.462328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.0.98.1:35461)
0929 00:56:43.462641 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:43.462653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:43.463050 (+ 397us) server_negotiation.cc:407] Connection header received
0929 00:56:43.463169 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:43.463174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:43.463358 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:43.463475 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:43.464095 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.464884 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.465652 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.466127 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.466294 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:43.468759 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:43.468767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:43.468778 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:43.468799 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:43.470468 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:43.470844 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:43.470848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:43.470849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:43.470932 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:43.471145 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:43.471147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:43.471148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:43.471345 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:43.471497 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:43.471620 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 00:56:43.471732 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":121,"threads_started":1}
W20260929 00:56:43.472736 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54398
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:56:43.497223 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:43.489345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.0.98.1:35461)
0929 00:56:43.489517 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:43.489521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:43.489582 (+ 61us) server_negotiation.cc:407] Connection header received
0929 00:56:43.489693 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:43.489695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:43.489736 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:43.489806 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:43.490564 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.491114 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.492094 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:43.492324 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:43.492404 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:43.494705 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:43.494708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:43.494710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:43.494730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:43.495886 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:43.496227 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:43.496230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:43.496231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:43.496296 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:43.496494 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:43.496496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:43.496497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:43.496637 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:43.496727 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:43.496902 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 00:56:43.496990 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 00:56:43.498165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 506
I20260929 00:56:43.504014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 640
I20260929 00:56:43.511564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 774
I20260929 00:56:43.517014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 412
2026-09-29T00:56:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5008 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[917](info): set up 2 sockets
Sep 29 00:56:43 dist-test-slave-30s6 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 00:56:44.545704 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:56:44 dist-test-slave-30s6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:56:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:56:44.662611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34097
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:44.767709 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:44.767968 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:44.768014 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:44.771930 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:44.772006 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:44.772027 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:44.772042 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:44.772058 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:44.777262 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:44.778226 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:44.779165 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:56:44.789143 930 server_base.cc:1034] running on GCE node
W20260929 00:56:44.789116 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:44.789110 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:44.789127 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:44.789940 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:44.790999 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:44.792179 930 hybrid_clock.cc:648] HybridClock initialized: now 1790643404792155 us; error 46 us; skew 500 ppm
Sep 29 00:56:44 dist-test-slave-30s6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:44.797693 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:56:44.798641 930 webserver.cc:460] Webserver started at http://127.0.98.62:38907/ using document root <none> and password file <none>
I20260929 00:56:44.799192 930 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:44.799279 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:44.799506 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:44.801494 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "49f9b7b63ffc4dd696033766fd220d81"
format_stamp: "Formatted at 2026-09-29 00:56:44 on dist-test-slave-30s6"
server_key: "c5941c14a31b970f03fa3234d259013d"
server_key_iv: "cde7b1007270c6c3c3601ed31508e019"
server_key_version: "encryptionkey@0"
I20260929 00:56:44.802052 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "49f9b7b63ffc4dd696033766fd220d81"
format_stamp: "Formatted at 2026-09-29 00:56:44 on dist-test-slave-30s6"
server_key: "c5941c14a31b970f03fa3234d259013d"
server_key_iv: "cde7b1007270c6c3c3601ed31508e019"
server_key_version: "encryptionkey@0"
I20260929 00:56:44.805545 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:56:44.807875 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:44.809062 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 00:56:44.809206 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "49f9b7b63ffc4dd696033766fd220d81"
format_stamp: "Formatted at 2026-09-29 00:56:44 on dist-test-slave-30s6"
server_key: "c5941c14a31b970f03fa3234d259013d"
server_key_iv: "cde7b1007270c6c3c3601ed31508e019"
server_key_version: "encryptionkey@0"
I20260929 00:56:44.809309 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:44.825953 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:44.828853 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:44.829048 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:44.835592 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34097
I20260929 00:56:44.835609 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34097 every 8 connection(s)
I20260929 00:56:44.836462 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:44.837842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 930
I20260929 00:56:44.837929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:56:44.838224 392 external_mini_cluster.cc:1428] Setting key efbe363e8931bd2529d0181ef8732b17
I20260929 00:56:44.839291 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:44.845351 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81: Bootstrap starting.
Sep 29 00:56:44 dist-test-slave-30s6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:44.848006 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:44.848667 998 log.cc:824] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:44.850874 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81: No bootstrap required, opened a new log
I20260929 00:56:44.852167 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:44.839554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.0.98.62:34097)
0929 00:56:44.839927 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:44.839942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:44.839977 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:56:44.840636 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:44.840652 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:44.840949 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:44.841218 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:44.842059 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:44.843055 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:44.844101 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:44.844502 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:44.844600 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:44.847408 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:44.847416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:44.847435 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:44.847472 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:44.849540 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:44.849967 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:44.849971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:44.849975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:44.850084 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:44.850369 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:44.850371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:44.850372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:44.850753 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:44.850893 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:44.851185 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 00:56:44.851474 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":120,"threads_started":1}
I20260929 00:56:44.854575 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9b7b63ffc4dd696033766fd220d81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260929 00:56:44.854780 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:44.854856 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f9b7b63ffc4dd696033766fd220d81, State: Initialized, Role: FOLLOWER
I20260929 00:56:44.855264 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9b7b63ffc4dd696033766fd220d81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260929 00:56:44.855396 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:56:44.855456 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:56:44.855566 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:44.856510 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9b7b63ffc4dd696033766fd220d81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260929 00:56:44.856830 998 leader_election.cc:302] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f9b7b63ffc4dd696033766fd220d81; no voters:
I20260929 00:56:44.857121 998 leader_election.cc:288] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:56:44.857239 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:44.857468 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [term 1 LEADER]: Becoming Leader. State: Replica: 49f9b7b63ffc4dd696033766fd220d81, State: Running, Role: LEADER
I20260929 00:56:44.857797 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9b7b63ffc4dd696033766fd220d81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260929 00:56:44.858378 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:44.859465 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f9b7b63ffc4dd696033766fd220d81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9b7b63ffc4dd696033766fd220d81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } }
I20260929 00:56:44.859598 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:44.859941 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f9b7b63ffc4dd696033766fd220d81. Latest consensus state: current_term: 1 leader_uuid: "49f9b7b63ffc4dd696033766fd220d81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9b7b63ffc4dd696033766fd220d81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } }
I20260929 00:56:44.860106 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:44.860688 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:56:44.863394 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:44.867851 1012 catalog_manager.cc:1347] Generated new cluster ID: b7a88fb953764ab880e5c0b72415e2b9
I20260929 00:56:44.867908 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:44.882735 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:44.883881 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:44.896754 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f9b7b63ffc4dd696033766fd220d81: Generated new TSK 0
I20260929 00:56:44.897387 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:56:44.925515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:45.032637 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:45.032903 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:45.032980 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:45.036883 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:45.036963 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:45.037065 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:56:45.042300 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:45.043308 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:45.044271 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:45.053370 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:45.053370 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:45.053387 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:45.053611 1024 server_base.cc:1034] running on GCE node
I20260929 00:56:45.054034 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:45.054665 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:45.055817 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790643405055799 us; error 30 us; skew 500 ppm
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:45.058440 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:56:45.059262 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35395/ using document root <none> and password file <none>
I20260929 00:56:45.059836 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:45.059896 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:45.060055 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:45.062029 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "e50f4b088b6b4883927f7cacc7b6a94c"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "984401a577c020f0450658c5b786cb3d"
server_key_iv: "a4416e044a7a5cdc2e25285173d0bb7e"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.062673 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e50f4b088b6b4883927f7cacc7b6a94c"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "984401a577c020f0450658c5b786cb3d"
server_key_iv: "a4416e044a7a5cdc2e25285173d0bb7e"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.065999 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:45.068295 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.069451 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:45.069548 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "e50f4b088b6b4883927f7cacc7b6a94c"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "984401a577c020f0450658c5b786cb3d"
server_key_iv: "a4416e044a7a5cdc2e25285173d0bb7e"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.069628 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:45.128692 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:45.131547 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:45.131724 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:45.132230 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:45.132972 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:45.133015 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.133052 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:45.133075 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.141269 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36855
I20260929 00:56:45.141300 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36855 every 8 connection(s)
I20260929 00:56:45.142100 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:45.151856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 1024
I20260929 00:56:45.151966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:56:45.152273 392 external_mini_cluster.cc:1428] Setting key b26e2b8f5dea0ada6f2c72ef9dace117
I20260929 00:56:45.155458 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.143899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33905 (local address 127.0.98.62:34097)
0929 00:56:45.144038 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.144042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.144875 (+ 833us) server_negotiation.cc:407] Connection header received
0929 00:56:45.146208 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.146212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.146267 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.146346 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:45.147502 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.148041 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.149176 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.149415 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.149529 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.151848 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:45.151852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:45.151855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:45.151885 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:45.153483 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.153954 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.153959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.153968 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.154063 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.154432 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.154437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.154439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.154612 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:45.154718 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.155187 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.155304 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:56:45.156260 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.144265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:33905)
0929 00:56:45.144712 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:45.144767 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:45.145980 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:45.146510 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:45.146517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:45.146843 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:45.147349 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.147361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.148166 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.148169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.149042 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.149051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.149564 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.149568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.149731 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.150293 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:45.150307 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:45.151689 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:45.153586 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:45.153591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:45.153606 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:45.153855 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:45.154157 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:45.154160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:45.154162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:45.154332 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:45.154714 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:45.154721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:45.154962 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 00:56:45.155151 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 00:56:45.155473 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":141,"threads_started":1}
I20260929 00:56:45.157405 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260929 00:56:45.157655 1153 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:45.158164 1153 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260929 00:56:45.159781 962 ts_manager.cc:194] Registered new tserver with Master: e50f4b088b6b4883927f7cacc7b6a94c (127.0.98.1:36855)
I20260929 00:56:45.160857 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33905
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:56:45.177255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:45.284659 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:45.284930 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:45.285013 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:45.288954 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:45.289011 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:45.289119 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:56:45.294121 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:45.295289 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:45.296324 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:45.305400 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:45.305437 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:45.305611 1158 server_base.cc:1034] running on GCE node
W20260929 00:56:45.305444 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:45.306085 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:45.306731 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:45.307884 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790643405307848 us; error 47 us; skew 500 ppm
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:45.310482 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:56:45.311275 1158 webserver.cc:460] Webserver started at http://127.0.98.2:33601/ using document root <none> and password file <none>
I20260929 00:56:45.311787 1158 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:45.311846 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:45.312067 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:45.314098 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "0695eacb6e754f82881ae713e61ec170"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "0e9f98f158dc32965a7d76cccead8d09"
server_key_iv: "5e93a8f19ca6525c7e4c86ae9da1c7ac"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.314687 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0695eacb6e754f82881ae713e61ec170"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "0e9f98f158dc32965a7d76cccead8d09"
server_key_iv: "5e93a8f19ca6525c7e4c86ae9da1c7ac"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.318110 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:45.320423 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.321557 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:45.321673 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "0695eacb6e754f82881ae713e61ec170"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "0e9f98f158dc32965a7d76cccead8d09"
server_key_iv: "5e93a8f19ca6525c7e4c86ae9da1c7ac"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.321779 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:45.333801 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:45.336382 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:45.336570 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:45.337111 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:45.337872 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:45.337924 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.337988 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:45.338029 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.346405 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39869
I20260929 00:56:45.346448 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39869 every 8 connection(s)
I20260929 00:56:45.347198 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:56:45.352495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 1158
I20260929 00:56:45.352597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:56:45.352847 392 external_mini_cluster.cc:1428] Setting key 24b5b2db72f618bc70575ce6e487a723
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:45.361569 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.348792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60849 (local address 127.0.98.62:34097)
0929 00:56:45.348966 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.348971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.349779 (+ 808us) server_negotiation.cc:407] Connection header received
0929 00:56:45.351061 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.351063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.351105 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.351174 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:45.352438 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.353172 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.354406 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.354788 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.354903 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.357559 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:45.357562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:45.357564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:45.357586 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:45.359254 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.359931 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.359933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.359935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.360008 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.360465 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.360467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.360468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.360678 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:45.360790 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.361359 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.361458 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:56:45.362717 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.349181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:60849)
0929 00:56:45.349634 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:45.349674 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:45.350880 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:45.351328 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:45.351334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:45.351847 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:45.352311 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.352323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.353295 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.353301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.354301 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.354309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.354936 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.354939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.355062 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.355645 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:45.355660 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:45.357374 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:45.359394 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:45.359420 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:45.359439 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:45.359830 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:45.360150 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:45.360156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:45.360160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:45.360354 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:45.360812 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:45.360819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:45.361112 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 00:56:45.361324 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 00:56:45.361657 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:56:45.363777 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260929 00:56:45.364027 1287 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:45.364485 1287 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:56:45.365486 962 ts_manager.cc:194] Registered new tserver with Master: 0695eacb6e754f82881ae713e61ec170 (127.0.98.2:39869)
I20260929 00:56:45.366321 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:56:45.381004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:45.486055 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:45.486351 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:45.486420 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:45.490203 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:45.490260 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:45.490312 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:56:45.495282 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:45.496264 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:45.497246 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:45.506130 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:45.506155 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:45.506294 1292 server_base.cc:1034] running on GCE node
W20260929 00:56:45.506130 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:45.506821 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:45.507422 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:45.508580 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790643405508563 us; error 28 us; skew 500 ppm
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:45.511332 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:56:45.512152 1292 webserver.cc:460] Webserver started at http://127.0.98.3:39711/ using document root <none> and password file <none>
I20260929 00:56:45.512696 1292 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:45.512782 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:45.513026 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:45.515014 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f826c9d671c456dac32a41c54af0cdd"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "bfaf082d4bba03cac2429b0ee5555e71"
server_key_iv: "a33bdab82a878fd6d9c47c999c13ba6b"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.515551 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f826c9d671c456dac32a41c54af0cdd"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "bfaf082d4bba03cac2429b0ee5555e71"
server_key_iv: "a33bdab82a878fd6d9c47c999c13ba6b"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.519055 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:45.521217 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.522392 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:56:45.522514 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "6f826c9d671c456dac32a41c54af0cdd"
format_stamp: "Formatted at 2026-09-29 00:56:45 on dist-test-slave-30s6"
server_key: "bfaf082d4bba03cac2429b0ee5555e71"
server_key_iv: "a33bdab82a878fd6d9c47c999c13ba6b"
server_key_version: "encryptionkey@0"
I20260929 00:56:45.522611 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:45.548947 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:45.552279 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:45.552464 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:45.552990 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:45.553716 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:45.553759 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.553794 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:45.553815 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:45.562156 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35015
I20260929 00:56:45.562209 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35015 every 8 connection(s)
I20260929 00:56:45.563006 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:56:45.566962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 1292
I20260929 00:56:45.567080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:56:45.567382 392 external_mini_cluster.cc:1428] Setting key 95852207619029e0e868b124cf7f745b
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:45.576006 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.564665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48507 (local address 127.0.98.62:34097)
0929 00:56:45.564803 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.564808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.565642 (+ 834us) server_negotiation.cc:407] Connection header received
0929 00:56:45.566966 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.566969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.567044 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.567138 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:45.568585 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.569111 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.570303 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.570523 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.570642 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.572852 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:45.572857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:45.572860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:45.572891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:45.574080 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.574548 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.574554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.574556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.574634 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.574976 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.574979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.574981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.575148 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:45.575247 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.575718 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.575836 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:56:45.576718 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.565089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:48507)
0929 00:56:45.565495 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:45.565535 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:45.566758 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:45.567341 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:45.567349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:45.567805 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:45.568416 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.568458 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.569232 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.569236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.570157 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.570166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.570672 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.570675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.570831 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.571302 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:45.571316 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:45.572664 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:45.574201 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:45.574206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:45.574218 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:45.574450 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:45.574739 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:45.574742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:45.574743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:45.574880 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:45.575256 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:45.575264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:45.575472 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 00:56:45.575648 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 00:56:45.575956 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260929 00:56:45.577714 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260929 00:56:45.577905 1421 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:45.578366 1421 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260929 00:56:45.579241 962 ts_manager.cc:194] Registered new tserver with Master: 6f826c9d671c456dac32a41c54af0cdd (127.0.98.3:35015)
I20260929 00:56:45.579841 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48507
I20260929 00:56:45.581044 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:56:45.590695 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.584292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.0.98.62:34097)
0929 00:56:45.584398 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.584401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.584587 (+ 186us) server_negotiation.cc:407] Connection header received
0929 00:56:45.584674 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.584676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.584708 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.584771 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:45.585502 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.585963 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.586769 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.586975 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.587041 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.588228 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:45.588230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:45.588232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:45.588246 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:45.589376 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.589721 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.589723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.589724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.589787 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.590002 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.590004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.590005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.590173 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:45.590330 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.590506 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.590582 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 00:56:45.594679 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:56:45.597453 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:45.611088 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.605084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39869 (local address 127.0.0.1:34090)
0929 00:56:45.605667 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:45.605715 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:45.605831 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:45.606972 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:45.606975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:45.606992 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:45.607137 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.607146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.608174 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.608181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.609622 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.609632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.610558 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.610561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.610670 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.610703 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:56:45.610806 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 00:56:45.610916 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":103,"thread_start_us":75,"threads_started":1}
I20260929 00:56:45.611289 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.605228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.0.98.2:39869)
0929 00:56:45.605792 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.605803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.605826 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:56:45.606490 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.606497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.606669 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.606821 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:45.607263 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.608050 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.609756 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.610439 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.610558 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.610700 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.611017 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.611116 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":105,"threads_started":1}
I20260929 00:56:45.611253 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.605755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36855 (local address 127.0.0.1:45168)
0929 00:56:45.606084 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:45.606098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:45.606209 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:45.606833 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:45.606839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:45.606876 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:45.607129 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.607136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.608569 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.608573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.609836 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.609844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.610817 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.610820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.610905 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.610923 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 00:56:45.611010 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 00:56:45.611068 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":215,"spinlock_wait_cycles":7424,"thread_start_us":74,"threads_started":1}
I20260929 00:56:45.611601 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.605540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.0.98.3:35015)
0929 00:56:45.605930 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.605941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.606155 (+ 214us) server_negotiation.cc:407] Connection header received
0929 00:56:45.606268 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.606274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.606475 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.606636 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:45.607235 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.608425 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.609830 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.610669 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.610789 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.610922 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.611324 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.611429 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":119,"threads_started":1}
I20260929 00:56:45.611678 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.606150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.0.98.1:36855)
0929 00:56:45.606448 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.606462 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.606482 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:56:45.606542 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.606546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.606683 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.606820 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:45.607288 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.608433 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.609952 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.610721 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.610817 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.610922 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.611424 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.611526 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
I20260929 00:56:45.611368 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.605165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.0.1:59484)
0929 00:56:45.605949 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:45.605961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:45.606043 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:45.606833 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:45.606839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:45.606874 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:45.607103 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.607113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.608612 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.608615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.609710 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.609719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.610995 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.610999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.611084 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.611106 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:56:45.611174 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 00:56:45.611239 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":80,"spinlock_wait_cycles":7040,"thread_start_us":95,"threads_started":1}
I20260929 00:56:45.614542 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 243110a7c0f94105b08af1ccc4ce407f (DEFAULT_TABLE table=test-table [id=ee2523be8bcd4d1c9f37d856bf2c3442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:45.614564 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 243110a7c0f94105b08af1ccc4ce407f (DEFAULT_TABLE table=test-table [id=ee2523be8bcd4d1c9f37d856bf2c3442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:45.614564 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 243110a7c0f94105b08af1ccc4ce407f (DEFAULT_TABLE table=test-table [id=ee2523be8bcd4d1c9f37d856bf2c3442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:45.615440 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 243110a7c0f94105b08af1ccc4ce407f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:45.615501 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 243110a7c0f94105b08af1ccc4ce407f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:45.615501 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 243110a7c0f94105b08af1ccc4ce407f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:45.621304 1438 tablet_bootstrap.cc:492] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd: Bootstrap starting.
I20260929 00:56:45.622376 1440 tablet_bootstrap.cc:492] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c: Bootstrap starting.
I20260929 00:56:45.623021 1439 tablet_bootstrap.cc:492] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170: Bootstrap starting.
I20260929 00:56:45.623690 1438 tablet_bootstrap.cc:654] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:45.624464 1440 tablet_bootstrap.cc:654] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:45.624804 1439 tablet_bootstrap.cc:654] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:45.625164 1438 log.cc:824] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:45.626056 1439 log.cc:824] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:45.626145 1440 log.cc:824] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:45.627724 1438 tablet_bootstrap.cc:492] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd: No bootstrap required, opened a new log
I20260929 00:56:45.627928 1439 tablet_bootstrap.cc:492] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170: No bootstrap required, opened a new log
I20260929 00:56:45.627918 1438 ts_tablet_manager.cc:1397] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 00:56:45.628095 1440 tablet_bootstrap.cc:492] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c: No bootstrap required, opened a new log
I20260929 00:56:45.628114 1439 ts_tablet_manager.cc:1397] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260929 00:56:45.628264 1440 ts_tablet_manager.cc:1397] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:56:45.630889 1439 raft_consensus.cc:348] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.631223 1439 raft_consensus.cc:374] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:45.631381 1439 raft_consensus.cc:729] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0695eacb6e754f82881ae713e61ec170, State: Initialized, Role: FOLLOWER
I20260929 00:56:45.631811 1439 consensus_queue.cc:260] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.631732 1440 raft_consensus.cc:348] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.631711 1438 raft_consensus.cc:348] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.631939 1440 raft_consensus.cc:374] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:45.631948 1438 raft_consensus.cc:374] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:45.631999 1438 raft_consensus.cc:729] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f826c9d671c456dac32a41c54af0cdd, State: Initialized, Role: FOLLOWER
I20260929 00:56:45.632025 1440 raft_consensus.cc:729] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50f4b088b6b4883927f7cacc7b6a94c, State: Initialized, Role: FOLLOWER
I20260929 00:56:45.632390 1287 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260929 00:56:45.632403 1438 consensus_queue.cc:260] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.632477 1440 consensus_queue.cc:260] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.632723 1439 ts_tablet_manager.cc:1428] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260929 00:56:45.633285 1438 ts_tablet_manager.cc:1428] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:56:45.633285 1440 ts_tablet_manager.cc:1428] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 00:56:45.633641 1153 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260929 00:56:45.633662 1421 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
W20260929 00:56:45.643153 1154 tablet.cc:2369] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:56:45.706732 1445 raft_consensus.cc:482] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:45.706930 1445 raft_consensus.cc:504] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.708035 1445 leader_election.cc:288] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0695eacb6e754f82881ae713e61ec170 (127.0.98.2:39869), e50f4b088b6b4883927f7cacc7b6a94c (127.0.98.1:36855)
I20260929 00:56:45.711822 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.708207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39869 (local address 127.0.98.3:50659)
0929 00:56:45.708388 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:45.708403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:45.708507 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:45.708797 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:45.708816 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:45.708842 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:45.708962 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.708971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.709757 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.709760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.710695 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.710703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.711369 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.711373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.711513 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.711553 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:56:45.711617 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 00:56:45.711668 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":34}
I20260929 00:56:45.711858 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.708287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50659 (local address 127.0.98.2:39869)
0929 00:56:45.708429 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.708433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.708454 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:56:45.708630 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.708633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.708694 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.708776 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:45.709081 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.709657 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.710814 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.711261 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.711340 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.711386 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.711652 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.711731 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:56:45.712020 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.708433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55549 (local address 127.0.98.1:36855)
0929 00:56:45.708561 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.708567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.708739 (+ 172us) server_negotiation.cc:407] Connection header received
0929 00:56:45.708871 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.708875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.708933 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.709010 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:45.709308 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.709813 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.710966 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.711365 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.711467 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.711527 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.711800 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.711879 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:56:45.712218 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.708267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36855 (local address 127.0.98.3:55549)
0929 00:56:45.708651 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:45.708664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:45.708765 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:45.709067 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:45.709070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:45.709086 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:45.709200 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.709206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.709914 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.709916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.710839 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:45.710848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.711474 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.711477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:45.711640 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.711664 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:56:45.711784 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 00:56:45.711843 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":64,"thread_start_us":110,"threads_started":1}
I20260929 00:56:45.712597 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243110a7c0f94105b08af1ccc4ce407f" candidate_uuid: "6f826c9d671c456dac32a41c54af0cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0695eacb6e754f82881ae713e61ec170" is_pre_election: true
I20260929 00:56:45.712827 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243110a7c0f94105b08af1ccc4ce407f" candidate_uuid: "6f826c9d671c456dac32a41c54af0cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" is_pre_election: true
I20260929 00:56:45.712919 1241 raft_consensus.cc:2454] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f826c9d671c456dac32a41c54af0cdd in term 0.
I20260929 00:56:45.713135 1107 raft_consensus.cc:2454] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f826c9d671c456dac32a41c54af0cdd in term 0.
I20260929 00:56:45.713439 1311 leader_election.cc:302] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0695eacb6e754f82881ae713e61ec170, 6f826c9d671c456dac32a41c54af0cdd; no voters:
I20260929 00:56:45.713757 1445 raft_consensus.cc:2781] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:45.713840 1445 raft_consensus.cc:482] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:45.713871 1445 raft_consensus.cc:3037] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:45.714905 1445 raft_consensus.cc:504] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.715246 1445 leader_election.cc:288] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [CANDIDATE]: Term 1 election: Requested vote from peers 0695eacb6e754f82881ae713e61ec170 (127.0.98.2:39869), e50f4b088b6b4883927f7cacc7b6a94c (127.0.98.1:36855)
I20260929 00:56:45.715476 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243110a7c0f94105b08af1ccc4ce407f" candidate_uuid: "6f826c9d671c456dac32a41c54af0cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0695eacb6e754f82881ae713e61ec170"
I20260929 00:56:45.715586 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243110a7c0f94105b08af1ccc4ce407f" candidate_uuid: "6f826c9d671c456dac32a41c54af0cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50f4b088b6b4883927f7cacc7b6a94c"
I20260929 00:56:45.715595 1241 raft_consensus.cc:3037] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:45.715719 1107 raft_consensus.cc:3037] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:45.716635 1241 raft_consensus.cc:2454] T 243110a7c0f94105b08af1ccc4ce407f P 0695eacb6e754f82881ae713e61ec170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f826c9d671c456dac32a41c54af0cdd in term 1.
I20260929 00:56:45.716691 1107 raft_consensus.cc:2454] T 243110a7c0f94105b08af1ccc4ce407f P e50f4b088b6b4883927f7cacc7b6a94c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f826c9d671c456dac32a41c54af0cdd in term 1.
I20260929 00:56:45.717020 1311 leader_election.cc:302] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0695eacb6e754f82881ae713e61ec170, 6f826c9d671c456dac32a41c54af0cdd; no voters:
I20260929 00:56:45.717190 1445 raft_consensus.cc:2781] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:45.717399 1445 raft_consensus.cc:686] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [term 1 LEADER]: Becoming Leader. State: Replica: 6f826c9d671c456dac32a41c54af0cdd, State: Running, Role: LEADER
I20260929 00:56:45.717720 1445 consensus_queue.cc:237] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } }
I20260929 00:56:45.720631 962 catalog_manager.cc:5495] T 243110a7c0f94105b08af1ccc4ce407f P 6f826c9d671c456dac32a41c54af0cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f826c9d671c456dac32a41c54af0cdd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6f826c9d671c456dac32a41c54af0cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0695eacb6e754f82881ae713e61ec170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f826c9d671c456dac32a41c54af0cdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e50f4b088b6b4883927f7cacc7b6a94c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36855 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:56:45.732710 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.724137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.0.98.1:36855)
0929 00:56:45.724263 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.724266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.724444 (+ 178us) server_negotiation.cc:407] Connection header received
0929 00:56:45.724526 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.724528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.724559 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.724616 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:45.725435 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.726013 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.726929 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.727130 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.727204 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.729457 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:45.729475 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:45.729489 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:45.729509 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:45.731034 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.731463 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.731475 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.731477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.731556 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.731830 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.731833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.731835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.732065 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:45.732287 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.732431 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.732548 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:56:45.758524 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.750782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.0.98.1:36855)
0929 00:56:45.750972 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.750976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.751036 (+ 60us) server_negotiation.cc:407] Connection header received
0929 00:56:45.751156 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.751159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.751198 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.751275 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:45.751928 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.752410 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.753304 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.753504 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.753593 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.755885 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:45.755888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:45.755890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:45.755912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:45.757069 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.757445 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.757447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.757448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.757528 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.757739 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.757741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.757742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.757881 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:45.757972 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.758305 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.758397 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:56:45.783017 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:45.775191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.0.98.1:36855)
0929 00:56:45.775368 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:45.775371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:45.775439 (+ 68us) server_negotiation.cc:407] Connection header received
0929 00:56:45.775529 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:45.775532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:45.775570 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:45.775639 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:45.776251 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.776757 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.777506 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:45.777719 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:45.777812 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:45.779978 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:45.779981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:45.779982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:45.780003 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:45.781355 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.781803 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.781808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.781809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.781952 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:45.782174 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:45.782176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:45.782212 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:45.782394 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:45.782492 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:45.782676 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 00:56:45.782760 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 00:56:45.784297 1067 tablet_service.cc:2908] Rejecting scan request for tablet 243110a7c0f94105b08af1ccc4ce407f: Uninitialized: clean time has not yet been initialized
I20260929 00:56:45.786520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 1024
I20260929 00:56:45.792397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 1158
I20260929 00:56:45.797456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 1292
I20260929 00:56:45.802655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 930
2026-09-29T00:56:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2286 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[1462](info): set up 2 sockets
Sep 29 00:56:45 dist-test-slave-30s6 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260929 00:56:46.813807 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:56:46 dist-test-slave-30s6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:56:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:56:46.925367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:47.032804 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:47.033085 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:47.033152 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:47.037114 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:47.037209 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:56:47.037238 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:47.037281 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:47.037317 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:47.037350 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:47.042313 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42773
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:47.043277 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:47.044245 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:47.052536 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:47.052707 1475 server_base.cc:1034] running on GCE node
W20260929 00:56:47.052536 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:47.052518 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:47.053483 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:47.054591 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:47.055771 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790643407055751 us; error 30 us; skew 500 ppm
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:47.058362 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:56:47.059193 1475 webserver.cc:460] Webserver started at http://127.0.98.62:35371/ using document root <none> and password file <none>
I20260929 00:56:47.059716 1475 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:47.059801 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:47.060014 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:47.061932 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "d04f5f8f168142bb88140e79c5747811"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "15adcf463cec4c47d8fecdad61d62600"
server_key_iv: "ccf66b756174480dfb83341050336323"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.062485 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "d04f5f8f168142bb88140e79c5747811"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "15adcf463cec4c47d8fecdad61d62600"
server_key_iv: "ccf66b756174480dfb83341050336323"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.065948 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:56:47.068351 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:47.069455 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:47.069581 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "d04f5f8f168142bb88140e79c5747811"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "15adcf463cec4c47d8fecdad61d62600"
server_key_iv: "ccf66b756174480dfb83341050336323"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.069679 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:47.088347 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:47.091027 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:47.091197 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:47.098513 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42773
I20260929 00:56:47.098529 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42773 every 8 connection(s)
I20260929 00:56:47.099370 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:47.100461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 1475
I20260929 00:56:47.100584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:56:47.100840 392 external_mini_cluster.cc:1428] Setting key 3f87e56c16c6666df2d4e7874bfc0c2a
I20260929 00:56:47.102988 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:47.108233 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811: Bootstrap starting.
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:47.110893 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:47.112171 1553 log.cc:824] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:47.114066 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811: No bootstrap required, opened a new log
I20260929 00:56:47.114089 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.101994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.0.98.62:42773)
0929 00:56:47.102380 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.102393 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.102427 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:56:47.103070 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.103095 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.103421 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.103763 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:47.104532 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.105463 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.106318 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.106754 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.106930 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.109604 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:47.109609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:47.109620 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:47.109645 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:47.111313 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.111820 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.111824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.111835 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.111934 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.112228 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.112230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.112231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.112684 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:47.112899 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.113124 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.113427 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
I20260929 00:56:47.117225 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04f5f8f168142bb88140e79c5747811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } }
I20260929 00:56:47.117443 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:47.117527 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04f5f8f168142bb88140e79c5747811, State: Initialized, Role: FOLLOWER
I20260929 00:56:47.117930 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04f5f8f168142bb88140e79c5747811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } }
I20260929 00:56:47.118067 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:56:47.118147 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:56:47.118269 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:47.119078 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04f5f8f168142bb88140e79c5747811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } }
I20260929 00:56:47.119390 1553 leader_election.cc:302] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04f5f8f168142bb88140e79c5747811; no voters:
I20260929 00:56:47.119663 1553 leader_election.cc:288] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:56:47.119745 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:47.119990 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [term 1 LEADER]: Becoming Leader. State: Replica: d04f5f8f168142bb88140e79c5747811, State: Running, Role: LEADER
I20260929 00:56:47.120321 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04f5f8f168142bb88140e79c5747811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } }
I20260929 00:56:47.120836 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:47.121843 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04f5f8f168142bb88140e79c5747811. Latest consensus state: current_term: 1 leader_uuid: "d04f5f8f168142bb88140e79c5747811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04f5f8f168142bb88140e79c5747811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } } }
I20260929 00:56:47.121891 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d04f5f8f168142bb88140e79c5747811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04f5f8f168142bb88140e79c5747811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } } }
I20260929 00:56:47.121954 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:47.121978 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:47.122339 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:56:47.124998 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:47.129890 1567 catalog_manager.cc:1347] Generated new cluster ID: 22d1fd5d8f2d4b2d9fd618b180302299
I20260929 00:56:47.129959 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:47.160317 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:47.161456 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:47.172576 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811: Generated new TSK 0
I20260929 00:56:47.173256 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:56:47.207129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42773
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:47.314424 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:47.314666 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:47.314706 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:47.318783 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:47.318845 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:47.318912 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:56:47.318988 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:56:47.324527 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:47.325570 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:47.326575 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:56:47.335541 1579 server_base.cc:1034] running on GCE node
W20260929 00:56:47.335552 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:47.335732 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:47.335731 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:47.336138 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:47.336702 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:47.337877 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790643407337867 us; error 29 us; skew 500 ppm
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:47.340449 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:56:47.341250 1579 webserver.cc:460] Webserver started at http://127.0.98.1:34899/ using document root <none> and password file <none>
I20260929 00:56:47.341773 1579 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:47.341859 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:47.342077 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:47.344004 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "de2fc61e5b41499088242b75a0775266"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "40cf868d84833970c76951b4f63fd18a"
server_key_iv: "11a378bde34b542220fb259dd184435f"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.344522 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "de2fc61e5b41499088242b75a0775266"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "40cf868d84833970c76951b4f63fd18a"
server_key_iv: "11a378bde34b542220fb259dd184435f"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.347913 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:56:47.350426 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:47.351511 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:56:47.351636 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "de2fc61e5b41499088242b75a0775266"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "40cf868d84833970c76951b4f63fd18a"
server_key_iv: "11a378bde34b542220fb259dd184435f"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.351737 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:47.368319 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:47.371007 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:47.371205 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:47.372601 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:47.372668 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:47.372735 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:47.372774 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:47.383504 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36895
I20260929 00:56:47.384236 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36895 every 8 connection(s)
I20260929 00:56:47.384351 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:56:47.386083 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.373354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43145 (local address 127.0.98.62:42773)
0929 00:56:47.373472 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.373476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.374358 (+ 882us) server_negotiation.cc:407] Connection header received
0929 00:56:47.375319 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.375323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.375387 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.375465 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:47.377227 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.378080 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.379286 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.379540 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.379653 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.382554 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:47.382557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:47.382559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:47.382582 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:47.384092 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.384653 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.384657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.384659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.384759 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.385130 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.385133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.385135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.385340 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:47.385425 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.385870 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.385971 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 00:56:47.386838 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.373767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42773 (local address 127.0.98.1:43145)
0929 00:56:47.374201 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.374248 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.375068 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.375948 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.375957 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.376468 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:47.377076 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.377096 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.378211 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.378214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.379164 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.379173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.379696 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.379700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.379868 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.380439 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:47.380462 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:47.382394 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:47.384222 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.384229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.384243 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.384552 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.384872 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.384875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.384877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.385027 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.385431 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:47.385436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:47.385658 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.385796 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.386029 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
I20260929 00:56:47.388059 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42773
I20260929 00:56:47.388294 1710 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:47.388798 1710 heartbeater.cc:511] Master 127.0.98.62:42773 requested a full tablet report, sending...
I20260929 00:56:47.390431 1506 ts_manager.cc:194] Registered new tserver with Master: de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895)
I20260929 00:56:47.391525 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43145
I20260929 00:56:47.393306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 1579
I20260929 00:56:47.393419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:56:47.393687 392 external_mini_cluster.cc:1428] Setting key 6ae5aca7aea9135aed437b9edc15fba0
I20260929 00:56:47.399541 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.390380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42773 (local address 127.0.98.1:56065)
0929 00:56:47.392114 (+ 1734us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.392125 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.392237 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.392480 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.392483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.392665 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:47.392755 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.392760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.393510 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.393514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.394155 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.394163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.394848 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.394851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.394963 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.395506 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:47.395520 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:47.395945 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:47.398040 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.398043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.398045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.398330 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.398628 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.398630 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.398631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.398700 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.399135 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:47.399137 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:47.399210 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.399289 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.399399 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1650,"thread_start_us":105,"threads_started":1}
I20260929 00:56:47.399734 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.390473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56065 (local address 127.0.98.62:42773)
0929 00:56:47.390630 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.390634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.392209 (+ 1575us) server_negotiation.cc:407] Connection header received
0929 00:56:47.392336 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.392339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.392383 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.392465 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:47.392868 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.393335 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.394311 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.394611 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.394707 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.396094 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:47.396100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:47.396106 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:47.396135 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:47.397901 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.398434 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.398441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.398445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.398533 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.398793 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.398799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.398803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.399043 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:47.399147 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.399372 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.399504 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:56:47.420886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42773
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:47.526472 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:47.526700 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:47.526741 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:47.530597 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:47.530638 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:47.530673 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:56:47.530704 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:56:47.535528 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:47.536473 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:47.537367 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:47.546376 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:47.546407 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:47.546382 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:47.546728 1719 server_base.cc:1034] running on GCE node
I20260929 00:56:47.547106 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:47.547704 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:47.548861 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790643407548843 us; error 29 us; skew 500 ppm
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:47.551427 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:56:47.552263 1719 webserver.cc:460] Webserver started at http://127.0.98.2:42203/ using document root <none> and password file <none>
I20260929 00:56:47.552778 1719 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:47.552832 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:47.553063 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:47.554977 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "4439ef00135b4dbb905daa919587daed"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "1a57b4d963763995ab8ae8128d8d188e"
server_key_iv: "b14a5cee257339f1a97bddf60655b864"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.555487 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4439ef00135b4dbb905daa919587daed"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "1a57b4d963763995ab8ae8128d8d188e"
server_key_iv: "b14a5cee257339f1a97bddf60655b864"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.559029 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:56:47.561383 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:47.562554 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:56:47.562676 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "4439ef00135b4dbb905daa919587daed"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "1a57b4d963763995ab8ae8128d8d188e"
server_key_iv: "b14a5cee257339f1a97bddf60655b864"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.562777 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:47.612094 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:47.615131 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:47.615327 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:47.616695 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:47.616778 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:47.616861 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:47.616899 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:47.627878 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43597
I20260929 00:56:47.628671 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:56:47.630348 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.617580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47071 (local address 127.0.98.62:42773)
0929 00:56:47.617725 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.617729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.618593 (+ 864us) server_negotiation.cc:407] Connection header received
0929 00:56:47.619622 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.619626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.619682 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.619728 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:47.621249 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.622154 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.623451 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.623749 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.623849 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.626691 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:47.626695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:47.626701 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:47.626728 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:47.628203 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.628689 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.628694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.628695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.628778 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.629275 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.629279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.629280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.629508 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:47.629606 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.630069 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.630162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:56:47.629446 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43597 every 8 connection(s)
I20260929 00:56:47.631102 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.617988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42773 (local address 127.0.98.2:47071)
0929 00:56:47.618436 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.618484 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.619372 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.620100 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.620106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.620453 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:47.621102 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.621123 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.622327 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.622331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.623331 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.623340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.623889 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.623893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.624064 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.624620 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:47.624647 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:47.626536 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:47.628314 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.628321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.628337 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.628588 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.628876 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.628880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.628882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.629031 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.629596 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:47.629601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:47.629863 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.630021 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.630300 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":162,"threads_started":1}
I20260929 00:56:47.632449 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42773
I20260929 00:56:47.632584 1850 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:47.633028 1850 heartbeater.cc:511] Master 127.0.98.62:42773 requested a full tablet report, sending...
I20260929 00:56:47.633936 1507 ts_manager.cc:194] Registered new tserver with Master: 4439ef00135b4dbb905daa919587daed (127.0.98.2:43597)
I20260929 00:56:47.634640 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47071
I20260929 00:56:47.637172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 1719
I20260929 00:56:47.637277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:56:47.637528 392 external_mini_cluster.cc:1428] Setting key 307d9ef3495c13bf81a0c238a7a732a4
I20260929 00:56:47.641331 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.634390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42773 (local address 127.0.98.2:37937)
0929 00:56:47.634691 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.634702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.634803 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.635049 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.635053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.635266 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:47.635376 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.635382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.636079 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.636082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.636635 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.636642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.637126 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.637129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.637197 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.637683 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:47.637694 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:47.638097 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:47.639816 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.639822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.639827 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.640087 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.640411 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.640413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.640415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.640486 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.640948 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:47.640950 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:47.641017 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.641091 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.641185 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260929 00:56:47.641523 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.634449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37937 (local address 127.0.98.62:42773)
0929 00:56:47.634564 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.634568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.634771 (+ 203us) server_negotiation.cc:407] Connection header received
0929 00:56:47.634898 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.634901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.634944 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.635033 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:47.635482 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.635961 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.636749 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.637017 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.637117 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.638243 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:47.638247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:47.638250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:47.638272 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:47.639706 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.640205 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.640209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.640212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.640307 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.640571 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.640576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.640578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.640850 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:47.640952 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.641176 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.641293 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:56:47.663375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42773
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:47.770562 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:47.770815 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:47.770907 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:47.774977 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:47.775030 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:47.775099 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:56:47.775193 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:56:47.780292 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:47.781312 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:47.782420 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:47.791882 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:47.791882 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:47.791879 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:47.792179 1859 server_base.cc:1034] running on GCE node
I20260929 00:56:47.792593 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:47.793236 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:47.794414 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790643407794401 us; error 30 us; skew 500 ppm
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:47.796924 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:56:47.797752 1859 webserver.cc:460] Webserver started at http://127.0.98.3:43941/ using document root <none> and password file <none>
I20260929 00:56:47.798399 1859 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:47.798485 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:47.798704 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:47.800619 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "141bfdb778de403f9ae1d991f22b4fb8"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "c00bf7c948a6a1b7b5ca84d6da95dd6f"
server_key_iv: "9ae341cdf212d1f2d51a32ddbd704010"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.801172 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "141bfdb778de403f9ae1d991f22b4fb8"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "c00bf7c948a6a1b7b5ca84d6da95dd6f"
server_key_iv: "9ae341cdf212d1f2d51a32ddbd704010"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.804677 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 00:56:47.806995 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:47.808127 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:47.808254 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "141bfdb778de403f9ae1d991f22b4fb8"
format_stamp: "Formatted at 2026-09-29 00:56:47 on dist-test-slave-30s6"
server_key: "c00bf7c948a6a1b7b5ca84d6da95dd6f"
server_key_iv: "9ae341cdf212d1f2d51a32ddbd704010"
server_key_version: "encryptionkey@0"
I20260929 00:56:47.808374 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:47.835866 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:47.839200 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:47.839402 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:47.840843 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:47.840910 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:47.840979 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:47.841028 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:47.852116 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43387
I20260929 00:56:47.852157 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43387 every 8 connection(s)
I20260929 00:56:47.853307 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:56:47.855729 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.841569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46829 (local address 127.0.98.62:42773)
0929 00:56:47.841714 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.841720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.842607 (+ 887us) server_negotiation.cc:407] Connection header received
0929 00:56:47.843961 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.843968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.844036 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.844102 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:47.847244 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.847913 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.849157 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.849401 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.849511 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.851995 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:47.852000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:47.852004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:47.852041 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:47.853663 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.854242 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.854246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.854247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.854348 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.854689 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.854692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.854694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.854877 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:47.854979 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.855435 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.855567 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:56:47.856468 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.841939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42773 (local address 127.0.98.3:46829)
0929 00:56:47.842422 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.842477 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.843654 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.845787 (+ 2133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.845799 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.846341 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:47.847075 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.847098 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.848066 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.848070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.849036 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.849045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.849551 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.849555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.849726 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.850306 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:47.850326 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:47.851813 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:47.853771 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.853783 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.853797 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.854097 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.854426 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.854431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.854433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.854588 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.854975 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:47.854980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:47.855206 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.855381 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.855665 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
I20260929 00:56:47.857575 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42773
I20260929 00:56:47.857726 1990 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:47.858160 1990 heartbeater.cc:511] Master 127.0.98.62:42773 requested a full tablet report, sending...
I20260929 00:56:47.859107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 1859
I20260929 00:56:47.859192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:56:47.859210 1507 ts_manager.cc:194] Registered new tserver with Master: 141bfdb778de403f9ae1d991f22b4fb8 (127.0.98.3:43387)
I20260929 00:56:47.859403 392 external_mini_cluster.cc:1428] Setting key ea21dde3628c8b9d9fe0aefcf0bff745
I20260929 00:56:47.859921 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46829
I20260929 00:56:47.861387 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:56:47.867179 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.859640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42773 (local address 127.0.98.3:33931)
0929 00:56:47.859903 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.859917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.859998 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.860327 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.860334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.860507 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:47.860615 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.860621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.861629 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.861632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.862362 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.862370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.862804 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.862807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.862894 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.863378 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:47.863389 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:47.863673 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:47.865546 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.865548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.865550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.865805 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.866123 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:47.866125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:47.866126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:47.866272 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:47.866692 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:47.866694 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:47.866798 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.866892 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.867044 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260929 00:56:47.867323 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.859749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33931 (local address 127.0.98.62:42773)
0929 00:56:47.860034 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.860039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.860054 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:56:47.860088 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.860091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.860135 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.860205 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:47.860745 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.861472 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.862473 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.862698 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.862811 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.863813 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:47.863819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:47.863823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:47.863852 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:47.865433 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.865930 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.865935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.865937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.866030 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.866361 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.866367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.866371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.866595 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:47.866700 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.866981 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.867091 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:56:47 dist-test-slave-30s6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:47.895735 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.887468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.0.98.62:42773)
0929 00:56:47.887624 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.887641 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.887660 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:56:47.887703 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.887706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.887773 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.887883 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:47.888868 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.889492 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.890678 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.890912 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.890988 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.893209 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:47.893212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:47.893214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:47.893235 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:47.894491 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.894827 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.894830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.894831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.894896 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:47.895093 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:47.895095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:47.895096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:47.895239 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:47.895358 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.895532 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.895618 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:56:47.898478 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:56:47.901413 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:47.915464 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.909447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.0.98.2:43597)
0929 00:56:47.909742 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.909754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.910247 (+ 493us) server_negotiation.cc:407] Connection header received
0929 00:56:47.910330 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.910338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.910548 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.910694 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:47.911181 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.912378 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.914013 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.914817 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.914924 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.915075 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.915226 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.915319 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
I20260929 00:56:47.915455 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.909257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43597 (local address 127.0.0.1:51366)
0929 00:56:47.909908 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.909955 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.910079 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.910894 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.910899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.910920 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:47.911034 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.911040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.912493 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.912496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.913886 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.913895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.914946 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.914951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.915049 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.915081 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.915189 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.915299 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":304,"thread_start_us":83,"threads_started":1}
I20260929 00:56:47.915944 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.910116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.0.98.3:43387)
0929 00:56:47.910601 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.910615 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.910989 (+ 374us) server_negotiation.cc:407] Connection header received
0929 00:56:47.911115 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.911122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.911292 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.911452 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:47.911675 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.912403 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.914313 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.915018 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.915113 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.915266 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.915719 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.915808 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":158,"threads_started":1}
I20260929 00:56:47.916002 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.909505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36895 (local address 127.0.0.1:44700)
0929 00:56:47.910671 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.910683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.910770 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.911137 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.911140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.911154 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:47.911250 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.911255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.912740 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.912743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.914265 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.914282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.915174 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.915180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.915267 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.915298 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.915362 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.915413 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":117,"thread_start_us":87,"threads_started":1}
I20260929 00:56:47.916513 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.909961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43387 (local address 127.0.0.1:39008)
0929 00:56:47.910898 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.910906 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.910993 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.911428 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.911431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.911446 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:47.911546 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.911591 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.912538 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.912540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.914174 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.914216 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.915334 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.915338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.915421 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.915443 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.916064 (+ 621us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.916397 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":109,"thread_start_us":75,"threads_started":1}
I20260929 00:56:47.916530 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.910363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.0.98.1:36895)
0929 00:56:47.910671 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.910677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.910773 (+ 96us) server_negotiation.cc:407] Connection header received
0929 00:56:47.910850 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.910855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.911023 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.911156 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:47.911357 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.912434 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.914388 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.914887 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.914987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.915115 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.915535 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.916402 (+ 867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260929 00:56:47.918510 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8fb1f08eb64f519a12d5bd6481071f (DEFAULT_TABLE table=test-table [id=95059d8fc08641e9bb81138e8117edf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:47.918874 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8fb1f08eb64f519a12d5bd6481071f (DEFAULT_TABLE table=test-table [id=95059d8fc08641e9bb81138e8117edf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:47.919013 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8fb1f08eb64f519a12d5bd6481071f (DEFAULT_TABLE table=test-table [id=95059d8fc08641e9bb81138e8117edf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:47.919587 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8fb1f08eb64f519a12d5bd6481071f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:47.919809 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8fb1f08eb64f519a12d5bd6481071f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:47.919891 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8fb1f08eb64f519a12d5bd6481071f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:47.925849 2013 tablet_bootstrap.cc:492] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: Bootstrap starting.
I20260929 00:56:47.927203 2014 tablet_bootstrap.cc:492] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: Bootstrap starting.
I20260929 00:56:47.927253 2015 tablet_bootstrap.cc:492] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: Bootstrap starting.
I20260929 00:56:47.928793 2015 tablet_bootstrap.cc:654] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:47.928828 2013 tablet_bootstrap.cc:654] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:47.929031 2014 tablet_bootstrap.cc:654] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:47.929574 2015 log.cc:824] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:47.929699 2013 log.cc:824] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:47.929699 2014 log.cc:824] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:47.931311 2015 tablet_bootstrap.cc:492] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: No bootstrap required, opened a new log
I20260929 00:56:47.931478 2013 tablet_bootstrap.cc:492] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: No bootstrap required, opened a new log
I20260929 00:56:47.931478 2014 tablet_bootstrap.cc:492] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: No bootstrap required, opened a new log
I20260929 00:56:47.931504 2015 ts_tablet_manager.cc:1397] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 00:56:47.931646 2013 ts_tablet_manager.cc:1397] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:56:47.931646 2014 ts_tablet_manager.cc:1397] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 00:56:47.934612 2015 raft_consensus.cc:348] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.934852 2015 raft_consensus.cc:374] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:47.934903 2015 raft_consensus.cc:729] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439ef00135b4dbb905daa919587daed, State: Initialized, Role: FOLLOWER
I20260929 00:56:47.935351 2015 consensus_queue.cc:260] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.935473 2013 raft_consensus.cc:348] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.935709 2013 raft_consensus.cc:374] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:47.935761 2013 raft_consensus.cc:729] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141bfdb778de403f9ae1d991f22b4fb8, State: Initialized, Role: FOLLOWER
I20260929 00:56:47.935942 2014 raft_consensus.cc:348] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.936121 1850 heartbeater.cc:503] Master 127.0.98.62:42773 was elected leader, sending a full tablet report...
I20260929 00:56:47.936154 2014 raft_consensus.cc:374] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:47.936203 2014 raft_consensus.cc:729] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2fc61e5b41499088242b75a0775266, State: Initialized, Role: FOLLOWER
I20260929 00:56:47.936175 2013 consensus_queue.cc:260] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.936335 2015 ts_tablet_manager.cc:1428] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:56:47.936571 2014 consensus_queue.cc:260] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.937093 2013 ts_tablet_manager.cc:1428] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:56:47.937129 1710 heartbeater.cc:503] Master 127.0.98.62:42773 was elected leader, sending a full tablet report...
I20260929 00:56:47.937350 2014 ts_tablet_manager.cc:1428] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:56:47.937398 1990 heartbeater.cc:503] Master 127.0.98.62:42773 was elected leader, sending a full tablet report...
I20260929 00:56:47.953332 2020 raft_consensus.cc:482] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:47.953511 2020 raft_consensus.cc:504] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.954618 2020 leader_election.cc:288] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4439ef00135b4dbb905daa919587daed (127.0.98.2:43597), de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895)
I20260929 00:56:47.958702 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.954907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55013 (local address 127.0.98.1:36895)
0929 00:56:47.955069 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.955074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.955390 (+ 316us) server_negotiation.cc:407] Connection header received
0929 00:56:47.955493 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.955496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.955544 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.955623 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:47.955915 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.956453 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.957525 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.958156 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.958270 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.958334 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.958522 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.958588 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:56:47.958717 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.954837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36895 (local address 127.0.98.3:55013)
0929 00:56:47.955287 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.955301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.955399 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.955646 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.955648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.955662 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:47.955811 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.955817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.956583 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.956587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.957405 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.957413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.958274 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.958277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.958403 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.958420 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.958507 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.958575 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":31,"thread_start_us":154,"threads_started":1}
I20260929 00:56:47.959328 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.954828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43597 (local address 127.0.98.3:35089)
0929 00:56:47.955033 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:47.955049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:47.955132 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:47.955423 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:47.955426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:47.955466 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:47.955614 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.955621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.956673 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.956675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.958092 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:47.958102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.958994 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.958997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:47.959107 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.959124 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:56:47.959183 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 00:56:47.959231 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 00:56:47.959405 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.954918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35089 (local address 127.0.98.2:43597)
0929 00:56:47.955079 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.955086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.955102 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:56:47.955249 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.955252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.955305 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.955388 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:47.955720 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.956566 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.958257 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.958888 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.958976 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.959031 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.959215 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.959283 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 00:56:47.959321 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8fb1f08eb64f519a12d5bd6481071f" candidate_uuid: "141bfdb778de403f9ae1d991f22b4fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2fc61e5b41499088242b75a0775266" is_pre_election: true
I20260929 00:56:47.959632 1664 raft_consensus.cc:2454] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 141bfdb778de403f9ae1d991f22b4fb8 in term 0.
I20260929 00:56:47.959944 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8fb1f08eb64f519a12d5bd6481071f" candidate_uuid: "141bfdb778de403f9ae1d991f22b4fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4439ef00135b4dbb905daa919587daed" is_pre_election: true
I20260929 00:56:47.960129 1876 leader_election.cc:302] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141bfdb778de403f9ae1d991f22b4fb8, de2fc61e5b41499088242b75a0775266; no voters:
I20260929 00:56:47.960204 1804 raft_consensus.cc:2454] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 141bfdb778de403f9ae1d991f22b4fb8 in term 0.
I20260929 00:56:47.960392 2020 raft_consensus.cc:2781] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:47.960479 2020 raft_consensus.cc:482] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:47.960533 2020 raft_consensus.cc:3037] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:47.961575 2020 raft_consensus.cc:504] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.961922 2020 leader_election.cc:288] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4439ef00135b4dbb905daa919587daed (127.0.98.2:43597), de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895)
I20260929 00:56:47.962234 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8fb1f08eb64f519a12d5bd6481071f" candidate_uuid: "141bfdb778de403f9ae1d991f22b4fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4439ef00135b4dbb905daa919587daed"
I20260929 00:56:47.962293 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8fb1f08eb64f519a12d5bd6481071f" candidate_uuid: "141bfdb778de403f9ae1d991f22b4fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2fc61e5b41499088242b75a0775266"
I20260929 00:56:47.962374 1804 raft_consensus.cc:3037] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:47.962405 1664 raft_consensus.cc:3037] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:47.963333 1804 raft_consensus.cc:2454] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 141bfdb778de403f9ae1d991f22b4fb8 in term 1.
I20260929 00:56:47.963392 1664 raft_consensus.cc:2454] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 141bfdb778de403f9ae1d991f22b4fb8 in term 1.
I20260929 00:56:47.963637 1878 leader_election.cc:302] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141bfdb778de403f9ae1d991f22b4fb8, 4439ef00135b4dbb905daa919587daed; no voters:
I20260929 00:56:47.963790 2020 raft_consensus.cc:2781] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:47.964005 2020 raft_consensus.cc:686] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 141bfdb778de403f9ae1d991f22b4fb8, State: Running, Role: LEADER
I20260929 00:56:47.964313 2020 consensus_queue.cc:237] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:47.967212 1506 catalog_manager.cc:5495] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 141bfdb778de403f9ae1d991f22b4fb8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "141bfdb778de403f9ae1d991f22b4fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } health_report { overall_health: HEALTHY } } }
I20260929 00:56:47.982194 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:47.979045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.0.98.3:43387)
0929 00:56:47.979176 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:47.979180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:47.979207 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:56:47.979381 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:47.979383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:47.979429 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:47.979506 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:47.979712 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.980178 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.980967 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:47.981167 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:47.981238 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:47.981494 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:47.981844 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:47.981922 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:47.981985 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 00:56:47.982026 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:56:47.987171 1804 raft_consensus.cc:1261] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Refusing update from remote peer 141bfdb778de403f9ae1d991f22b4fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:47.987182 1664 raft_consensus.cc:1261] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Refusing update from remote peer 141bfdb778de403f9ae1d991f22b4fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:47.987805 2023 consensus_queue.cc:1035] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:47.987990 2020 consensus_queue.cc:1035] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:47.994889 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334475399109525504 to 7334475399024349184. Current Snapshot: MvccSnapshot[applied={T|T < 7334475399109525504}]
I20260929 00:56:47.996438 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334475399109525504 to 7334475399024349184. Current Snapshot: MvccSnapshot[applied={T|T < 7334475399109525504}]
I20260929 00:56:48.006043 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:48.006243 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:48.008965 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811: Sending DeleteTablet for 3 replicas of tablet 1c8fb1f08eb64f519a12d5bd6481071f
I20260929 00:56:48.009706 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8fb1f08eb64f519a12d5bd6481071f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51366
I20260929 00:56:48.009699 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8fb1f08eb64f519a12d5bd6481071f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39008
I20260929 00:56:48.009739 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8fb1f08eb64f519a12d5bd6481071f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44700
I20260929 00:56:48.010228 2036 tablet_replica.cc:330] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: stopping tablet replica
I20260929 00:56:48.010535 2036 raft_consensus.cc:2229] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:56:48.010776 2037 tablet_replica.cc:330] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: stopping tablet replica
I20260929 00:56:48.010949 2036 raft_consensus.cc:2258] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:48.011060 2037 raft_consensus.cc:2229] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:48.011340 2037 raft_consensus.cc:2258] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:48.011297 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:56:48.011675 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:48.012410 2036 ts_tablet_manager.cc:1905] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:48.012437 2037 ts_tablet_manager.cc:1905] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:48.015429 2037 ts_tablet_manager.cc:1918] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:56:48.015523 2037 log.cc:1196] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/1c8fb1f08eb64f519a12d5bd6481071f
I20260929 00:56:48.015854 2037 ts_tablet_manager.cc:1939] T 1c8fb1f08eb64f519a12d5bd6481071f P de2fc61e5b41499088242b75a0775266: Deleting consensus metadata
I20260929 00:56:48.016662 1784 tablet_service.cc:1467] Processing CreateTablet for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 (DEFAULT_TABLE table=test-table [id=abd5f89c3f5547ca8472cc5eb95cfce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:48.016736 1924 tablet_service.cc:1467] Processing CreateTablet for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 (DEFAULT_TABLE table=test-table [id=abd5f89c3f5547ca8472cc5eb95cfce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:48.016934 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:48.016970 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:48.017123 2035 tablet_replica.cc:330] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: stopping tablet replica
I20260929 00:56:48.017387 2035 raft_consensus.cc:2229] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:48.017632 2035 raft_consensus.cc:2258] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:48.019063 2013 tablet_bootstrap.cc:492] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: Bootstrap starting.
I20260929 00:56:48.019405 2015 tablet_bootstrap.cc:492] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: Bootstrap starting.
I20260929 00:56:48.019691 1644 tablet_service.cc:1467] Processing CreateTablet for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 (DEFAULT_TABLE table=test-table [id=abd5f89c3f5547ca8472cc5eb95cfce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:48.019917 2036 ts_tablet_manager.cc:1918] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:56:48.019935 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:48.019996 2013 tablet_bootstrap.cc:654] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:48.020426 2015 tablet_bootstrap.cc:654] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:48.019997 2036 log.cc:1196] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/1c8fb1f08eb64f519a12d5bd6481071f
I20260929 00:56:48.021793 2036 ts_tablet_manager.cc:1939] T 1c8fb1f08eb64f519a12d5bd6481071f P 141bfdb778de403f9ae1d991f22b4fb8: Deleting consensus metadata
I20260929 00:56:48.022065 2015 tablet_bootstrap.cc:492] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: No bootstrap required, opened a new log
I20260929 00:56:48.022111 2014 tablet_bootstrap.cc:492] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: Bootstrap starting.
I20260929 00:56:48.022137 2015 ts_tablet_manager.cc:1397] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:56:48.022270 1492 catalog_manager.cc:4841] TS de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895): tablet 1c8fb1f08eb64f519a12d5bd6481071f (table test-table [id=95059d8fc08641e9bb81138e8117edf3]) successfully deleted
I20260929 00:56:48.022574 2015 raft_consensus.cc:348] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.022707 2015 raft_consensus.cc:374] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:48.022877 1492 catalog_manager.cc:4841] TS 141bfdb778de403f9ae1d991f22b4fb8 (127.0.98.3:43387): tablet 1c8fb1f08eb64f519a12d5bd6481071f (table test-table [id=95059d8fc08641e9bb81138e8117edf3]) successfully deleted
I20260929 00:56:48.023136 2015 raft_consensus.cc:729] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439ef00135b4dbb905daa919587daed, State: Initialized, Role: FOLLOWER
I20260929 00:56:48.023268 2015 consensus_queue.cc:260] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.023516 2014 tablet_bootstrap.cc:654] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:48.023938 2013 tablet_bootstrap.cc:492] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: No bootstrap required, opened a new log
I20260929 00:56:48.024009 2013 ts_tablet_manager.cc:1397] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 00:56:48.024322 2035 ts_tablet_manager.cc:1905] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:48.024410 2013 raft_consensus.cc:348] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.024497 2013 raft_consensus.cc:374] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:48.024525 2013 raft_consensus.cc:729] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141bfdb778de403f9ae1d991f22b4fb8, State: Initialized, Role: FOLLOWER
I20260929 00:56:48.024638 2014 tablet_bootstrap.cc:492] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: No bootstrap required, opened a new log
I20260929 00:56:48.024642 2013 consensus_queue.cc:260] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.024713 2014 ts_tablet_manager.cc:1397] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:56:48.024884 2013 ts_tablet_manager.cc:1428] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:48.025429 2014 raft_consensus.cc:348] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.025532 2014 raft_consensus.cc:374] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:48.025571 2014 raft_consensus.cc:729] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2fc61e5b41499088242b75a0775266, State: Initialized, Role: FOLLOWER
I20260929 00:56:48.025700 2014 consensus_queue.cc:260] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.025985 2014 ts_tablet_manager.cc:1428] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:48.026464 2015 ts_tablet_manager.cc:1428] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 00:56:48.027565 2035 ts_tablet_manager.cc:1918] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:56:48.027653 2035 log.cc:1196] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/1c8fb1f08eb64f519a12d5bd6481071f
I20260929 00:56:48.027968 2035 ts_tablet_manager.cc:1939] T 1c8fb1f08eb64f519a12d5bd6481071f P 4439ef00135b4dbb905daa919587daed: Deleting consensus metadata
I20260929 00:56:48.029057 1494 catalog_manager.cc:4841] TS 4439ef00135b4dbb905daa919587daed (127.0.98.2:43597): tablet 1c8fb1f08eb64f519a12d5bd6481071f (table test-table [id=95059d8fc08641e9bb81138e8117edf3]) successfully deleted
I20260929 00:56:48.037591 2019 raft_consensus.cc:482] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:48.037695 2019 raft_consensus.cc:504] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.038676 2019 leader_election.cc:288] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895), 141bfdb778de403f9ae1d991f22b4fb8 (127.0.98.3:43387)
I20260929 00:56:48.042974 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.039450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50407 (local address 127.0.98.3:43387)
0929 00:56:48.039544 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:48.039548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:48.039561 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:56:48.039602 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:48.039605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:48.039643 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:48.039721 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:48.040059 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.040685 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.041718 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.042331 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.042433 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.042505 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:48.042792 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 00:56:48.042863 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 00:56:48.042974 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.038926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36895 (local address 127.0.98.2:44965)
0929 00:56:48.039059 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:48.039076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:48.039162 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:48.039517 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:48.039519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:48.039536 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:48.039631 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.039636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.040509 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.040512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.041732 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.041742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.042657 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.042660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.042764 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.042781 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:56:48.042841 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 00:56:48.042894 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 00:56:48.043270 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.039012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44965 (local address 127.0.98.1:36895)
0929 00:56:48.039123 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:48.039127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:48.039140 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:56:48.039291 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:48.039294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:48.039338 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:48.039404 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:48.039735 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.040402 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.041888 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.042552 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.042653 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.042706 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:48.043040 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 00:56:48.043140 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 00:56:48.042982 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.039166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43387 (local address 127.0.98.2:50407)
0929 00:56:48.039398 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:48.039410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:48.039470 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:48.039825 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:48.039828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:48.039843 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:48.039949 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.039955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.040792 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.040796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.041600 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.041607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.042504 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.042507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.042622 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.042647 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:56:48.042752 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:56:48.042838 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260929 00:56:48.043618 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc21c5d0fbb940c8abfc4a1cf2c0e790" candidate_uuid: "4439ef00135b4dbb905daa919587daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2fc61e5b41499088242b75a0775266" is_pre_election: true
I20260929 00:56:48.043797 1664 raft_consensus.cc:2454] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4439ef00135b4dbb905daa919587daed in term 0.
I20260929 00:56:48.044039 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc21c5d0fbb940c8abfc4a1cf2c0e790" candidate_uuid: "4439ef00135b4dbb905daa919587daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141bfdb778de403f9ae1d991f22b4fb8" is_pre_election: true
I20260929 00:56:48.044117 1736 leader_election.cc:302] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4439ef00135b4dbb905daa919587daed, de2fc61e5b41499088242b75a0775266; no voters:
I20260929 00:56:48.044251 1944 raft_consensus.cc:2454] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4439ef00135b4dbb905daa919587daed in term 0.
I20260929 00:56:48.044332 2019 raft_consensus.cc:2781] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:48.044392 2019 raft_consensus.cc:482] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:48.044441 2019 raft_consensus.cc:3037] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:48.045238 2019 raft_consensus.cc:504] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.045532 2019 leader_election.cc:288] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [CANDIDATE]: Term 1 election: Requested vote from peers de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895), 141bfdb778de403f9ae1d991f22b4fb8 (127.0.98.3:43387)
I20260929 00:56:48.045871 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc21c5d0fbb940c8abfc4a1cf2c0e790" candidate_uuid: "4439ef00135b4dbb905daa919587daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2fc61e5b41499088242b75a0775266"
I20260929 00:56:48.045981 1664 raft_consensus.cc:3037] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:48.046842 1664 raft_consensus.cc:2454] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4439ef00135b4dbb905daa919587daed in term 1.
I20260929 00:56:48.047097 1736 leader_election.cc:302] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4439ef00135b4dbb905daa919587daed, de2fc61e5b41499088242b75a0775266; no voters:
I20260929 00:56:48.047088 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc21c5d0fbb940c8abfc4a1cf2c0e790" candidate_uuid: "4439ef00135b4dbb905daa919587daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141bfdb778de403f9ae1d991f22b4fb8"
I20260929 00:56:48.047217 1944 raft_consensus.cc:3037] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:48.048054 1944 raft_consensus.cc:2454] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4439ef00135b4dbb905daa919587daed in term 1.
I20260929 00:56:48.048245 2019 raft_consensus.cc:2781] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:48.048483 2019 raft_consensus.cc:686] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 1 LEADER]: Becoming Leader. State: Replica: 4439ef00135b4dbb905daa919587daed, State: Running, Role: LEADER
I20260929 00:56:48.048722 2019 consensus_queue.cc:237] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.050810 1507 catalog_manager.cc:5495] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed reported cstate change: term changed from 0 to 1, leader changed from <none> to 4439ef00135b4dbb905daa919587daed (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:56:48 dist-test-slave-30s6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:48.088994 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.079962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42773 (local address 127.0.0.1:51316)
0929 00:56:48.080243 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:48.080259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:48.080364 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:48.080601 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:48.080603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:48.081008 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:48.081096 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.081102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.082167 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.082169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.082795 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.082803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.083372 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.083374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.083434 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.084010 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:48.084022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:48.085578 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:48.087489 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:48.087496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:48.087500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:48.087787 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:48.088048 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:48.088050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:48.088051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:48.088126 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:48.088500 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:48.088505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:48.088598 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 00:56:48.088688 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 00:56:48.088854 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260929 00:56:48.089130 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.080040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.0.98.62:42773)
0929 00:56:48.080174 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:48.080179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:48.080345 (+ 166us) server_negotiation.cc:407] Connection header received
0929 00:56:48.080435 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:48.080438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:48.080484 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:48.080573 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:48.081211 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.082046 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.082908 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.083268 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.083366 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.085707 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:48.085711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:48.085714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:48.085737 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:48.087381 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:48.087872 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:48.087876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:48.087878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:48.087960 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:48.088208 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:48.088211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:48.088213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:48.088405 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:48.088500 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:48.088777 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 00:56:48.088934 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":54}
I20260929 00:56:48.092762 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51316:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:56:48.093472 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:48.098033 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 412277fb77e54b7e916431d4a21d6804 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15e612bd1b9340a68d7c3a895b13d1cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:56:48.098109 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 412277fb77e54b7e916431d4a21d6804 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15e612bd1b9340a68d7c3a895b13d1cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:56:48.098455 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412277fb77e54b7e916431d4a21d6804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:48.098361 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 412277fb77e54b7e916431d4a21d6804 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15e612bd1b9340a68d7c3a895b13d1cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:56:48.098524 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412277fb77e54b7e916431d4a21d6804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:48.098664 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412277fb77e54b7e916431d4a21d6804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:48.101200 2013 tablet_bootstrap.cc:492] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: Bootstrap starting.
I20260929 00:56:48.101332 2014 tablet_bootstrap.cc:492] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: Bootstrap starting.
I20260929 00:56:48.102125 2013 tablet_bootstrap.cc:654] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:48.102234 2014 tablet_bootstrap.cc:654] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:48.103269 2015 tablet_bootstrap.cc:492] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: Bootstrap starting.
I20260929 00:56:48.103459 2013 tablet_bootstrap.cc:492] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: No bootstrap required, opened a new log
I20260929 00:56:48.103541 2013 ts_tablet_manager.cc:1397] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:48.104004 2013 raft_consensus.cc:348] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.104137 2013 raft_consensus.cc:374] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:48.104164 2015 tablet_bootstrap.cc:654] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:48.104179 2013 raft_consensus.cc:729] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141bfdb778de403f9ae1d991f22b4fb8, State: Initialized, Role: FOLLOWER
I20260929 00:56:48.104313 2013 consensus_queue.cc:260] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.104604 2013 ts_tablet_manager.cc:1428] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260929 00:56:48.104682 1991 tablet.cc:2369] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:56:48.105613 2014 tablet_bootstrap.cc:492] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: No bootstrap required, opened a new log
I20260929 00:56:48.105732 2014 ts_tablet_manager.cc:1397] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 00:56:48.106096 2020 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.106264 2020 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:48.106202 2014 raft_consensus.cc:348] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.106315 2014 raft_consensus.cc:374] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:48.106359 2014 raft_consensus.cc:729] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2fc61e5b41499088242b75a0775266, State: Initialized, Role: FOLLOWER
I20260929 00:56:48.106482 2014 consensus_queue.cc:260] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.106778 2014 ts_tablet_manager.cc:1428] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 00:56:48.106990 2021 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.107122 2021 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:48.107380 2015 tablet_bootstrap.cc:492] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: No bootstrap required, opened a new log
I20260929 00:56:48.107470 2015 ts_tablet_manager.cc:1397] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 00:56:48.107853 2015 raft_consensus.cc:348] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.107963 2015 raft_consensus.cc:374] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:48.108001 2015 raft_consensus.cc:729] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439ef00135b4dbb905daa919587daed, State: Initialized, Role: FOLLOWER
I20260929 00:56:48.108130 2015 consensus_queue.cc:260] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.108443 2015 ts_tablet_manager.cc:1428] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:48.108609 2019 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.108723 2019 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 00:56:48.129931 1851 tablet.cc:2369] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:56:48.135673 1711 tablet.cc:2369] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:56:48.297900 2019 raft_consensus.cc:482] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:48.298034 2019 raft_consensus.cc:504] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.298442 2019 leader_election.cc:288] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895), 141bfdb778de403f9ae1d991f22b4fb8 (127.0.98.3:43387)
I20260929 00:56:48.298969 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412277fb77e54b7e916431d4a21d6804" candidate_uuid: "4439ef00135b4dbb905daa919587daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141bfdb778de403f9ae1d991f22b4fb8" is_pre_election: true
I20260929 00:56:48.298969 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412277fb77e54b7e916431d4a21d6804" candidate_uuid: "4439ef00135b4dbb905daa919587daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2fc61e5b41499088242b75a0775266" is_pre_election: true
I20260929 00:56:48.299124 1944 raft_consensus.cc:2454] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4439ef00135b4dbb905daa919587daed in term 0.
I20260929 00:56:48.299124 1664 raft_consensus.cc:2454] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4439ef00135b4dbb905daa919587daed in term 0.
I20260929 00:56:48.299412 1736 leader_election.cc:302] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141bfdb778de403f9ae1d991f22b4fb8, 4439ef00135b4dbb905daa919587daed; no voters:
I20260929 00:56:48.299572 2019 raft_consensus.cc:2781] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:48.299629 2019 raft_consensus.cc:482] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:48.299670 2019 raft_consensus.cc:3037] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:48.300587 2019 raft_consensus.cc:504] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.300904 2019 leader_election.cc:288] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [CANDIDATE]: Term 1 election: Requested vote from peers de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895), 141bfdb778de403f9ae1d991f22b4fb8 (127.0.98.3:43387)
I20260929 00:56:48.301249 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412277fb77e54b7e916431d4a21d6804" candidate_uuid: "4439ef00135b4dbb905daa919587daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2fc61e5b41499088242b75a0775266"
I20260929 00:56:48.301280 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412277fb77e54b7e916431d4a21d6804" candidate_uuid: "4439ef00135b4dbb905daa919587daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141bfdb778de403f9ae1d991f22b4fb8"
I20260929 00:56:48.301379 1664 raft_consensus.cc:3037] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:48.301379 1944 raft_consensus.cc:3037] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:48.302515 1664 raft_consensus.cc:2454] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4439ef00135b4dbb905daa919587daed in term 1.
I20260929 00:56:48.302515 1944 raft_consensus.cc:2454] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4439ef00135b4dbb905daa919587daed in term 1.
I20260929 00:56:48.302780 1736 leader_election.cc:302] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141bfdb778de403f9ae1d991f22b4fb8, 4439ef00135b4dbb905daa919587daed; no voters:
I20260929 00:56:48.302968 2019 raft_consensus.cc:2781] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:48.303082 2019 raft_consensus.cc:686] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [term 1 LEADER]: Becoming Leader. State: Replica: 4439ef00135b4dbb905daa919587daed, State: Running, Role: LEADER
I20260929 00:56:48.303211 2019 consensus_queue.cc:237] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } }
I20260929 00:56:48.303884 2042 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: TxnStatusTablet state changed. Reason: New leader 4439ef00135b4dbb905daa919587daed. Latest consensus state: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.303998 2042 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:56:48.304219 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:56:48.304418 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:56:48.304955 1507 catalog_manager.cc:5495] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed reported cstate change: term changed from 0 to 1, leader changed from <none> to 4439ef00135b4dbb905daa919587daed (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } health_report { overall_health: UNKNOWN } } }
I20260929 00:56:48.344036 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.340078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43597 (local address 127.0.0.1:51378)
0929 00:56:48.340228 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:48.340255 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:48.340361 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:48.340692 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:48.340695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:48.340702 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:56:48.340820 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.340825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.341527 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.341530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.342111 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.342118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.342614 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.342618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.342732 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.342780 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:56:48.343651 (+ 871us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:56:48.343668 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:56:48.343761 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:56:48.343893 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 00:56:48.344376 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.340178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.0.98.2:43597)
0929 00:56:48.340316 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:48.340320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:48.340334 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:56:48.340522 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:48.340525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:48.340579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:48.340679 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:48.340924 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.341421 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.342245 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.342459 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.342593 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.342947 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:48.343525 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:48.343654 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:48.344208 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 00:56:48.344263 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:56:48.351719 1664 raft_consensus.cc:1261] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Refusing update from remote peer 4439ef00135b4dbb905daa919587daed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:48.351783 1944 raft_consensus.cc:1261] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Refusing update from remote peer 4439ef00135b4dbb905daa919587daed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:48.352375 2019 consensus_queue.cc:1035] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [LEADER]: Connected to new peer: Peer: permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:48.352552 2042 consensus_queue.cc:1035] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:48.353904 2021 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: TxnStatusTablet state changed. Reason: New leader 4439ef00135b4dbb905daa919587daed. Latest consensus state: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.354007 2021 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:48.354957 2020 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: TxnStatusTablet state changed. Reason: New leader 4439ef00135b4dbb905daa919587daed. Latest consensus state: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.355063 2020 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:48.356971 2019 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.357064 2019 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:56:48.357157 2019 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.357213 2019 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P 4439ef00135b4dbb905daa919587daed: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:56:48.359921 2020 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.360020 2020 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P 141bfdb778de403f9ae1d991f22b4fb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:48.361171 2021 tablet_replica.cc:439] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4439ef00135b4dbb905daa919587daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 } } peers { permanent_uuid: "4439ef00135b4dbb905daa919587daed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 } } }
I20260929 00:56:48.361269 2021 tablet_replica.cc:442] T 412277fb77e54b7e916431d4a21d6804 P de2fc61e5b41499088242b75a0775266: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:48.366370 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.363581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.98.2:43597)
0929 00:56:48.363698 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:48.363702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:48.363713 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:56:48.363758 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:48.363760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:48.363794 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:48.363875 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:48.364092 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.364573 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.365362 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.365578 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.365680 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.365940 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:48.366023 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:48.366101 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:48.366209 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 00:56:48.366253 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:56:48.376242 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.373072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39059 (local address 127.0.98.2:43597)
0929 00:56:48.373197 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:48.373201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:48.373342 (+ 141us) server_negotiation.cc:407] Connection header received
0929 00:56:48.373494 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:48.373498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:48.373545 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:48.373621 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:48.373854 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.374354 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.375184 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.375403 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.375510 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.375826 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:48.375887 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:48.375958 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:48.376033 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 00:56:48.376103 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":38}
I20260929 00:56:48.376262 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.372994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43597 (local address 127.0.98.2:39059)
0929 00:56:48.373261 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:48.373274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:48.373396 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:48.373626 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:48.373629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:48.373635 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:56:48.373758 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.373763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.374444 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.374450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.375076 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:48.375083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.375507 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.375510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:48.375655 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.375699 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:56:48.375958 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:56:48.375964 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:56:48.376027 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 00:56:48.376122 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":43,"thread_start_us":95,"threads_started":1}
I20260929 00:56:48.382905 1664 raft_consensus.cc:1261] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Refusing update from remote peer 4439ef00135b4dbb905daa919587daed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:48.382905 1944 raft_consensus.cc:1261] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Refusing update from remote peer 4439ef00135b4dbb905daa919587daed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:48.383399 2019 consensus_queue.cc:1035] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2fc61e5b41499088242b75a0775266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:48.383704 2019 consensus_queue.cc:1035] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [LEADER]: Connected to new peer: Peer: permanent_uuid: "141bfdb778de403f9ae1d991f22b4fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:48.394356 2026 mvcc.cc:205] Tried to move back new op lower bound from 7334475400732266496 to 7334475399370436608. Current Snapshot: MvccSnapshot[applied={T|T < 7334475400732266496}]
I20260929 00:56:48.429870 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:48.430042 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:48.431950 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d04f5f8f168142bb88140e79c5747811: Sending DeleteTablet for 3 replicas of tablet dc21c5d0fbb940c8abfc4a1cf2c0e790
I20260929 00:56:48.432497 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44700
I20260929 00:56:48.432605 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51366
I20260929 00:56:48.432631 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39008
I20260929 00:56:48.432770 2035 tablet_replica.cc:330] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: stopping tablet replica
I20260929 00:56:48.432775 2037 tablet_replica.cc:330] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: stopping tablet replica
I20260929 00:56:48.432792 2036 tablet_replica.cc:330] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: stopping tablet replica
I20260929 00:56:48.432884 2037 raft_consensus.cc:2229] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:48.432899 2035 raft_consensus.cc:2229] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:56:48.432904 2036 raft_consensus.cc:2229] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:48.433010 2037 raft_consensus.cc:2258] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:48.433046 2036 raft_consensus.cc:2258] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:48.433179 2035 raft_consensus.cc:2258] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:48.433946 2035 ts_tablet_manager.cc:1905] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:48.433947 2037 ts_tablet_manager.cc:1905] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:48.434067 2036 ts_tablet_manager.cc:1905] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:48.436522 2035 ts_tablet_manager.cc:1918] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:48.436590 2035 log.cc:1196] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/dc21c5d0fbb940c8abfc4a1cf2c0e790
I20260929 00:56:48.436830 2035 ts_tablet_manager.cc:1939] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 4439ef00135b4dbb905daa919587daed: Deleting consensus metadata
I20260929 00:56:48.437286 2036 ts_tablet_manager.cc:1918] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:48.437348 2036 log.cc:1196] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/dc21c5d0fbb940c8abfc4a1cf2c0e790
I20260929 00:56:48.437570 2036 ts_tablet_manager.cc:1939] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P 141bfdb778de403f9ae1d991f22b4fb8: Deleting consensus metadata
I20260929 00:56:48.437676 1494 catalog_manager.cc:4841] TS 4439ef00135b4dbb905daa919587daed (127.0.98.2:43597): tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 (table test-table [id=abd5f89c3f5547ca8472cc5eb95cfce7]) successfully deleted
I20260929 00:56:48.438148 2037 ts_tablet_manager.cc:1918] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:48.438244 2037 log.cc:1196] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/dc21c5d0fbb940c8abfc4a1cf2c0e790
I20260929 00:56:48.438484 2037 ts_tablet_manager.cc:1939] T dc21c5d0fbb940c8abfc4a1cf2c0e790 P de2fc61e5b41499088242b75a0775266: Deleting consensus metadata
I20260929 00:56:48.439160 1492 catalog_manager.cc:4841] TS de2fc61e5b41499088242b75a0775266 (127.0.98.1:36895): tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 (table test-table [id=abd5f89c3f5547ca8472cc5eb95cfce7]) successfully deleted
I20260929 00:56:48.439478 1492 catalog_manager.cc:4841] TS 141bfdb778de403f9ae1d991f22b4fb8 (127.0.98.3:43387): tablet dc21c5d0fbb940c8abfc4a1cf2c0e790 (table test-table [id=abd5f89c3f5547ca8472cc5eb95cfce7]) successfully deleted
Sep 29 00:56:48 dist-test-slave-30s6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643407, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:56:48.447527 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.437853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.0.98.1:36895)
0929 00:56:48.437966 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:48.437974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:48.438451 (+ 477us) server_negotiation.cc:407] Connection header received
0929 00:56:48.438595 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:48.438598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:48.438659 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:48.438757 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:48.439652 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.440166 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.440958 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.441184 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.441288 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.444264 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:48.444274 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:48.444293 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:48.444313 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:48.445851 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:48.446299 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:48.446311 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:48.446313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:48.446383 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:48.446643 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:48.446647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:48.446648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:48.446881 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:48.447086 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:48.447257 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 00:56:48.447368 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 00:56:48.447902 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44706
I20260929 00:56:48.456512 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:48.449485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.0.98.62:42773)
0929 00:56:48.449610 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:48.449614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:48.449656 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:56:48.449760 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:48.449763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:48.449809 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:48.449890 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:48.450473 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.451250 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.451954 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:48.452263 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:48.452366 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:48.453512 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:48.453515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:48.453518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:48.453542 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:48.455079 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:48.455425 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:48.455428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:48.455429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:48.455526 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:48.455753 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:48.455756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:48.455758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:48.455936 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:48.456052 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:48.456239 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 00:56:48.456355 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 00:56:48.456837 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51326
I20260929 00:56:48.458014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 1579
I20260929 00:56:48.466689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 1719
I20260929 00:56:48.473388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 1859
I20260929 00:56:48.479632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 1475
2026-09-29T00:56:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2676 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:56:48 dist-test-slave-30s6 krb5kdc[2073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:56:48 dist-test-slave-30s6 krb5kdc[2073](info): set up 2 sockets
Sep 29 00:56:48 dist-test-slave-30s6 krb5kdc[2073](info): commencing operation
krb5kdc: starting...
W20260929 00:56:49.521894 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:56:49 dist-test-slave-30s6 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:56:49.636411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:49.740058 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:49.740289 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:49.740327 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:49.744241 2086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:49.744289 2086 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:56:49.744305 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:49.744323 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:49.744338 2086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:49.744352 2086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:49.749357 2086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35797
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:49.750669 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:49.751613 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:49.759529 2091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:49.759555 2094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:49.759572 2092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:49.759858 2086 server_base.cc:1034] running on GCE node
I20260929 00:56:49.760474 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:49.761600 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:49.762799 2086 hybrid_clock.cc:648] HybridClock initialized: now 1790643409762781 us; error 37 us; skew 500 ppm
Sep 29 00:56:49 dist-test-slave-30s6 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:49.765354 2086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:56:49.766108 2086 webserver.cc:460] Webserver started at http://127.0.98.62:39303/ using document root <none> and password file <none>
I20260929 00:56:49.766634 2086 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:49.766691 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:49.766837 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:49.768630 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "492c9800fd5a4bf4a552aade2d18cfdf"
format_stamp: "Formatted at 2026-09-29 00:56:49 on dist-test-slave-30s6"
server_key: "cba0e160346534cde2e4a59d5b478dd6"
server_key_iv: "33377c9b6e73494dbe30eae16bf42470"
server_key_version: "encryptionkey@0"
I20260929 00:56:49.769141 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "492c9800fd5a4bf4a552aade2d18cfdf"
format_stamp: "Formatted at 2026-09-29 00:56:49 on dist-test-slave-30s6"
server_key: "cba0e160346534cde2e4a59d5b478dd6"
server_key_iv: "33377c9b6e73494dbe30eae16bf42470"
server_key_version: "encryptionkey@0"
I20260929 00:56:49.772743 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 00:56:49.775034 2101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:49.776160 2086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 00:56:49.776283 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "492c9800fd5a4bf4a552aade2d18cfdf"
format_stamp: "Formatted at 2026-09-29 00:56:49 on dist-test-slave-30s6"
server_key: "cba0e160346534cde2e4a59d5b478dd6"
server_key_iv: "33377c9b6e73494dbe30eae16bf42470"
server_key_version: "encryptionkey@0"
I20260929 00:56:49.776384 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:49.788009 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:49.790551 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:49.790735 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:49.798002 2086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35797
I20260929 00:56:49.798022 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35797 every 8 connection(s)
I20260929 00:56:49.798897 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:49.801344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 2086
I20260929 00:56:49.801432 2164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:49.801455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:56:49.801683 392 external_mini_cluster.cc:1428] Setting key e18acb4a1e4f1ee7c8ce8fb7716da7fc
I20260929 00:56:49.808178 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: Bootstrap starting.
I20260929 00:56:49.810379 2164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:49.810988 2164 log.cc:824] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:49.812613 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: No bootstrap required, opened a new log
Sep 29 00:56:49 dist-test-slave-30s6 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:49.816082 2164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492c9800fd5a4bf4a552aade2d18cfdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } }
I20260929 00:56:49.816354 2164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:49.816425 2164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492c9800fd5a4bf4a552aade2d18cfdf, State: Initialized, Role: FOLLOWER
I20260929 00:56:49.816840 2164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492c9800fd5a4bf4a552aade2d18cfdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } }
I20260929 00:56:49.816955 2164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:56:49.816995 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:56:49.817054 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:49.817840 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492c9800fd5a4bf4a552aade2d18cfdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } }
I20260929 00:56:49.818166 2164 leader_election.cc:302] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 492c9800fd5a4bf4a552aade2d18cfdf; no voters:
I20260929 00:56:49.818512 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:56:49.818576 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:49.818797 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [term 1 LEADER]: Becoming Leader. State: Replica: 492c9800fd5a4bf4a552aade2d18cfdf, State: Running, Role: LEADER
I20260929 00:56:49.819133 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492c9800fd5a4bf4a552aade2d18cfdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } }
I20260929 00:56:49.819728 2164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:49.820917 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492c9800fd5a4bf4a552aade2d18cfdf. Latest consensus state: current_term: 1 leader_uuid: "492c9800fd5a4bf4a552aade2d18cfdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492c9800fd5a4bf4a552aade2d18cfdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } } }
I20260929 00:56:49.820971 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "492c9800fd5a4bf4a552aade2d18cfdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492c9800fd5a4bf4a552aade2d18cfdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } } }
I20260929 00:56:49.821127 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:49.821133 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:49.821602 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:56:49.822666 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:49.802975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.0.98.62:35797)
0929 00:56:49.803357 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:49.803375 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:49.803415 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:56:49.804085 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:49.804106 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:49.804497 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:49.804855 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:49.805455 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:49.806920 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:49.807741 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:49.808391 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:49.808555 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:49.816662 (+ 8107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:49.816670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:49.816688 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:49.816721 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:49.818619 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:49.819181 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:49.819187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:49.819203 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:49.819325 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:49.819598 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:49.819600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:49.819601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:49.820014 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:49.820230 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:49.820951 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 00:56:49.821245 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260929 00:56:49.824420 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:49.829617 2178 catalog_manager.cc:1347] Generated new cluster ID: 7022740106b543629aada54ab65d21e3
I20260929 00:56:49.829674 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:49.849050 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:49.849874 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:49.857272 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: Generated new TSK 0
I20260929 00:56:49.857829 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:56:49.889058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:49.993880 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:49.994107 2190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:49.994225 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:49.998098 2190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:49.998152 2190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:49.998210 2190 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:56:49.998226 2190 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:56:49.998281 2190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:56:50.003408 2190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:50.004417 2190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:50.005405 2190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:50.014487 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:50.014592 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:50.014712 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:50.014931 2190 server_base.cc:1034] running on GCE node
I20260929 00:56:50.015255 2190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:50.015829 2190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:50.016981 2190 hybrid_clock.cc:648] HybridClock initialized: now 1790643410016951 us; error 43 us; skew 500 ppm
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:50.019654 2190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:56:50.020546 2190 webserver.cc:460] Webserver started at http://127.0.98.1:37637/ using document root <none> and password file <none>
I20260929 00:56:50.021044 2190 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:50.021098 2190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:50.021319 2190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:50.023236 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "b706f745217546d3be5b4da860a15bc4"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "7ec4e4ed9576aaff39ce44caf3b724d0"
server_key_iv: "54ea0da10d80622516271d5e7052b533"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.023768 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b706f745217546d3be5b4da860a15bc4"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "7ec4e4ed9576aaff39ce44caf3b724d0"
server_key_iv: "54ea0da10d80622516271d5e7052b533"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.027081 2190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:56:50.029211 2205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:50.030403 2190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:56:50.030520 2190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "b706f745217546d3be5b4da860a15bc4"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "7ec4e4ed9576aaff39ce44caf3b724d0"
server_key_iv: "54ea0da10d80622516271d5e7052b533"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.030617 2190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:50.054307 2190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:50.057107 2190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:50.057296 2190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:50.058792 2190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:50.058912 2190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:50.058979 2190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:50.059002 2190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:50.069911 2190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41635
I20260929 00:56:50.069988 2320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41635 every 8 connection(s)
I20260929 00:56:50.070966 2190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:56:50.072150 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.059365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48955 (local address 127.0.98.62:35797)
0929 00:56:50.059488 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.059492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.060479 (+ 987us) server_negotiation.cc:407] Connection header received
0929 00:56:50.061418 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.061421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.061480 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.061584 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:50.062878 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.063389 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.064691 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.064977 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.065090 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.068354 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:50.068357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:50.068360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:50.068393 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:50.070154 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.070635 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.070640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.070642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.070733 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.071036 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.071039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.071041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.071255 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:50.071362 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.071913 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.072033 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:56:50.072857 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.059891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.1:48955)
0929 00:56:50.060337 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.060376 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.061206 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.061763 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.061771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.062279 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:50.062720 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.062734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.063509 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.063513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.064572 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.064581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.065123 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.065127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.065290 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.065900 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:50.065943 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:50.068196 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:50.070268 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.070279 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.070290 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.070515 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.070821 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.070823 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.070824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.070938 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.071374 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:50.071379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:50.071632 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.071839 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.072065 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":130,"threads_started":1}
I20260929 00:56:50.074304 2321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35797
I20260929 00:56:50.074437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 2190
I20260929 00:56:50.074520 2321 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:50.074548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:56:50.074828 392 external_mini_cluster.cc:1428] Setting key 54eecec7bf5c80d513e46ee0d99d0efa
I20260929 00:56:50.075069 2321 heartbeater.cc:511] Master 127.0.98.62:35797 requested a full tablet report, sending...
I20260929 00:56:50.076660 2118 ts_manager.cc:194] Registered new tserver with Master: b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635)
I20260929 00:56:50.078243 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48955
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:56:50.085529 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.076582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.1:56427)
0929 00:56:50.078287 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.078301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.078422 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.078737 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.078743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.078956 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:50.079118 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.079126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.080222 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.080225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.080880 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.080889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.081486 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.081489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.081552 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.082063 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:50.082084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:50.082443 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:50.084190 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.084192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.084194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.084366 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.084647 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.084649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.084650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.084715 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.085076 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:50.085078 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:50.085182 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.085279 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.085393 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1622,"thread_start_us":85,"threads_started":1}
I20260929 00:56:50.085602 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.076631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56427 (local address 127.0.98.62:35797)
0929 00:56:50.076738 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.076742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.078363 (+ 1621us) server_negotiation.cc:407] Connection header received
0929 00:56:50.078518 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.078521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.078559 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.078605 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:50.079239 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.080079 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.080967 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.081352 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.081485 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.082570 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:50.082573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:50.082576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:50.082600 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:50.084094 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.084459 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.084463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.084465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.084546 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.084797 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.084802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.084806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.084992 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:50.085091 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.085342 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.085492 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:56:50.101945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:50.209286 2330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:50.209535 2330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:50.209585 2330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:50.213501 2330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:50.213549 2330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:50.213586 2330 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:56:50.213599 2330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:56:50.213637 2330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:56:50.218595 2330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:50.219506 2330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:50.220484 2330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:50.229364 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:50.229384 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:50.229348 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:50.229449 2330 server_base.cc:1034] running on GCE node
I20260929 00:56:50.229887 2330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:50.230515 2330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:50.231721 2330 hybrid_clock.cc:648] HybridClock initialized: now 1790643410231710 us; error 29 us; skew 500 ppm
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:50.234431 2330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:56:50.235232 2330 webserver.cc:460] Webserver started at http://127.0.98.2:43165/ using document root <none> and password file <none>
I20260929 00:56:50.235782 2330 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:50.235869 2330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:50.236086 2330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:50.237990 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "0daffb0824884f4190a8d26143ef33b6"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "91c8b5e65794bfc37c915a89f2f04f31"
server_key_iv: "eef77a285b11c195fedbf11b234cb9ca"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.238579 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0daffb0824884f4190a8d26143ef33b6"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "91c8b5e65794bfc37c915a89f2f04f31"
server_key_iv: "eef77a285b11c195fedbf11b234cb9ca"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.242056 2330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:50.244254 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:50.245388 2330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:50.245510 2330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "0daffb0824884f4190a8d26143ef33b6"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "91c8b5e65794bfc37c915a89f2f04f31"
server_key_iv: "eef77a285b11c195fedbf11b234cb9ca"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.245610 2330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:50.271999 2330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:50.275336 2330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:50.275530 2330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:50.277000 2330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:50.277073 2330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:50.277141 2330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:50.277196 2330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:50.287952 2330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32945
I20260929 00:56:50.288182 2460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32945 every 8 connection(s)
I20260929 00:56:50.288854 2330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:56:50.291051 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.277876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43587 (local address 127.0.98.62:35797)
0929 00:56:50.277996 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.278000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.279077 (+ 1077us) server_negotiation.cc:407] Connection header received
0929 00:56:50.280165 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.280172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.280258 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.280321 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:50.282029 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.282938 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.284288 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.284683 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.284798 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.287437 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:50.287443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:50.287447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:50.287479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:50.289201 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.289779 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.289782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.289784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.289857 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.290137 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.290139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.290140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.290360 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:50.290436 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.290849 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.290941 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:56:50.291833 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.278455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.2:43587)
0929 00:56:50.278920 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.278961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.279914 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.280758 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.280769 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.281207 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:50.281851 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.281870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.283072 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.283076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.284163 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.284172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.284864 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.284871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.285051 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.285632 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:50.285646 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:50.287274 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:50.289346 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.289351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.289365 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.289681 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.289940 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.289942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.289943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.290063 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.290446 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:50.290450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:50.290661 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.290805 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.291024 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":134,"threads_started":1}
I20260929 00:56:50.292935 2461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35797
I20260929 00:56:50.293071 2461 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:50.293511 2461 heartbeater.cc:511] Master 127.0.98.62:35797 requested a full tablet report, sending...
I20260929 00:56:50.294569 2117 ts_manager.cc:194] Registered new tserver with Master: 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945)
I20260929 00:56:50.295356 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43587
I20260929 00:56:50.297923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 2330
I20260929 00:56:50.298036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:56:50.298370 392 external_mini_cluster.cc:1428] Setting key bbe29fcc7dbe95e956bb70a3d8da651b
I20260929 00:56:50.301733 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.295071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35379 (local address 127.0.98.62:35797)
0929 00:56:50.295185 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.295189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.295330 (+ 141us) server_negotiation.cc:407] Connection header received
0929 00:56:50.295401 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.295404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.295447 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.295520 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:50.295977 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.296464 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.297243 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.297469 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.297570 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.298734 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:50.298738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:50.298741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:50.298762 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:50.300144 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.300549 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.300554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.300556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.300633 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.300952 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.300958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.300961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.301189 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:50.301265 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.301546 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.301630 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:56:50.301864 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.294979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.2:35379)
0929 00:56:50.295255 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.295267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.295341 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.295631 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.295633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.295784 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:50.295867 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.295871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.296582 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.296586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.297133 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.297140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.297583 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.297587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.297686 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.298237 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:50.298249 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:50.298605 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:50.300262 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.300265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.300267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.300455 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.300723 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.300727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.300729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.300857 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.301281 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:50.301285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:50.301389 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.301501 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.301656 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:56:50.323212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:50.430600 2470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:50.430859 2470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:50.430928 2470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:50.434937 2470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:50.434999 2470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:50.435061 2470 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:56:50.435101 2470 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:56:50.435176 2470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:56:50.440282 2470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:50.441259 2470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:50.442237 2470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:50.451398 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:50.451406 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:50.451570 2470 server_base.cc:1034] running on GCE node
W20260929 00:56:50.451416 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:50.451970 2470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:50.452543 2470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:50.453692 2470 hybrid_clock.cc:648] HybridClock initialized: now 1790643410453655 us; error 49 us; skew 500 ppm
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:50.456282 2470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:56:50.457031 2470 webserver.cc:460] Webserver started at http://127.0.98.3:44917/ using document root <none> and password file <none>
I20260929 00:56:50.457513 2470 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:50.457567 2470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:50.457746 2470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:50.459690 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "c206d894043048c9b85dfec92de3eb6a"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "33298cc25fd502cde756e7ab161983fb"
server_key_iv: "067a97e34ddae5374dec03de3afab1f5"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.460177 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c206d894043048c9b85dfec92de3eb6a"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "33298cc25fd502cde756e7ab161983fb"
server_key_iv: "067a97e34ddae5374dec03de3afab1f5"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.463572 2470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:56:50.465736 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:50.466920 2470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:50.467046 2470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "c206d894043048c9b85dfec92de3eb6a"
format_stamp: "Formatted at 2026-09-29 00:56:50 on dist-test-slave-30s6"
server_key: "33298cc25fd502cde756e7ab161983fb"
server_key_iv: "067a97e34ddae5374dec03de3afab1f5"
server_key_version: "encryptionkey@0"
I20260929 00:56:50.467150 2470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:50.502337 2470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:50.505002 2470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:50.505185 2470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:50.506626 2470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:50.506690 2470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:50.506754 2470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:50.506786 2470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:50.517344 2470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35615
I20260929 00:56:50.517675 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35615 every 8 connection(s)
I20260929 00:56:50.518344 2470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:56:50.519152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 2470
I20260929 00:56:50.519302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:56:50.519644 392 external_mini_cluster.cc:1428] Setting key 1903a6e875ff28e7cd7ccd813c33a9d1
I20260929 00:56:50.520630 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.507290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55475 (local address 127.0.98.62:35797)
0929 00:56:50.507423 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.507427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.508303 (+ 876us) server_negotiation.cc:407] Connection header received
0929 00:56:50.509321 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.509326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.509389 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.509449 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:50.510936 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.511561 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.512764 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.513137 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.513265 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.516643 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:50.516647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:50.516649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:50.516673 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:50.518331 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.518911 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.518914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.518916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.519000 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.519448 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.519452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.519454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.519697 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:50.519792 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.520388 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.520509 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:56:50.521515 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.507716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.3:55475)
0929 00:56:50.508140 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.508187 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.509080 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.509898 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.509906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.510354 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:50.510800 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.510811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.511686 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.511690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.512641 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.512657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.513304 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.513308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.513526 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.514091 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:50.514116 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:50.516471 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:50.518434 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.518456 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.518469 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.518812 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.519086 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.519089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.519091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.519301 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.519845 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:50.519851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:50.520152 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.520340 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.520646 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":125,"threads_started":1}
I20260929 00:56:50.522735 2601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35797
I20260929 00:56:50.522891 2601 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:50.523505 2601 heartbeater.cc:511] Master 127.0.98.62:35797 requested a full tablet report, sending...
I20260929 00:56:50.524477 2117 ts_manager.cc:194] Registered new tserver with Master: c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615)
I20260929 00:56:50.525226 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55475
I20260929 00:56:50.531332 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.524930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34947 (local address 127.0.98.62:35797)
0929 00:56:50.525074 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.525076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.525175 (+ 99us) server_negotiation.cc:407] Connection header received
0929 00:56:50.525259 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.525261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.525292 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.525379 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:50.525771 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.526326 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.527074 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.527296 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.527396 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.528385 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:50.528388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:50.528391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:50.528412 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:50.529809 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.530282 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.530285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.530287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.530362 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.530618 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.530621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.530623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.530775 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:50.530869 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.531106 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.531203 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 00:56:50.531343 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.524826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.3:34947)
0929 00:56:50.525081 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.525101 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.525187 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.525394 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.525396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.525577 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:50.525666 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.525671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.526435 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.526439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.526966 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.526973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.527400 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.527403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.527486 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.527959 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:50.527972 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:50.528257 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:50.529927 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.529930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.529932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.530129 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.530448 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.530452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.530454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.530533 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.530870 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:50.530873 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:50.530957 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.531050 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.531181 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260929 00:56:50.534850 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:50.558077 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.550898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.0.98.62:35797)
0929 00:56:50.551008 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.551011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.551022 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:56:50.551052 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.551054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.551087 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.551153 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:50.551742 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.552202 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.552925 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.553112 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.553179 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.555343 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:50.555346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:50.555348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:50.555367 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:50.556696 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.557066 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.557070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.557072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.557164 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.557384 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.557386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.557387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.557531 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:50.557628 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.557803 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.557883 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:56:50.560691 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:56:50.563558 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:50.577229 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.571837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41635 (local address 127.0.0.1:56460)
0929 00:56:50.572127 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.572138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.572210 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.573075 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.573078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.573092 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:50.573254 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.573263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.574460 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.574465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.575939 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.575949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.576743 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.576746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.576840 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.576873 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.576972 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.577079 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":141,"thread_start_us":88,"threads_started":1}
I20260929 00:56:50.577423 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.571273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.0.98.3:35615)
0929 00:56:50.572513 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.572526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.572548 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:56:50.572622 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.572626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.572814 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.572959 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:50.573330 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.574560 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.575915 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.576729 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.576829 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.576988 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.577167 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.577267 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":96,"threads_started":1}
I20260929 00:56:50.577507 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.571272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.0.1:35690)
0929 00:56:50.572044 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.572057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.572199 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.572952 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.572958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.572994 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:50.573141 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.573148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.574718 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.574721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.575942 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.575949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.577096 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.577099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.577201 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.577221 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.577287 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.577344 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":32,"spinlock_wait_cycles":2432,"thread_start_us":79,"threads_started":1}
I20260929 00:56:50.577785 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.571147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35615 (local address 127.0.0.1:41394)
0929 00:56:50.571931 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.571978 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.572192 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.572953 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.572959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.572989 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:50.573222 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.573231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.574677 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.574680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.575792 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.575801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.576954 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.576960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.577044 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.577062 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.577129 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.577188 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":35,"spinlock_wait_cycles":1280,"thread_start_us":64,"threads_started":1}
I20260929 00:56:50.577906 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.571643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.0.98.2:32945)
0929 00:56:50.572015 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.572026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.572355 (+ 329us) server_negotiation.cc:407] Connection header received
0929 00:56:50.572438 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.572443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.572654 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.572806 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:50.573265 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.574623 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.576065 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.576830 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.576913 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.577034 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.577695 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.577767 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":79,"threads_started":1}
I20260929 00:56:50.578792 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.572210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.0.98.1:41635)
0929 00:56:50.572655 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.572672 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.572692 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:56:50.572760 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.572765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.572935 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.573158 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:50.573383 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.574309 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.576093 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.576614 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.576746 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.576882 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.577933 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.578700 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":98,"threads_started":1}
I20260929 00:56:50.580689 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 840c20622b9243c39b1b63b21dd20090 (DEFAULT_TABLE table=test-table [id=482bd3dd9e9f45eebcfa2dcc37f417b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:50.580696 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 840c20622b9243c39b1b63b21dd20090 (DEFAULT_TABLE table=test-table [id=482bd3dd9e9f45eebcfa2dcc37f417b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:50.581143 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 840c20622b9243c39b1b63b21dd20090 (DEFAULT_TABLE table=test-table [id=482bd3dd9e9f45eebcfa2dcc37f417b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:50.581635 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840c20622b9243c39b1b63b21dd20090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:50.581665 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840c20622b9243c39b1b63b21dd20090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:50.581995 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840c20622b9243c39b1b63b21dd20090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:50.587841 2624 tablet_bootstrap.cc:492] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Bootstrap starting.
I20260929 00:56:50.588604 2625 tablet_bootstrap.cc:492] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Bootstrap starting.
I20260929 00:56:50.589761 2626 tablet_bootstrap.cc:492] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Bootstrap starting.
I20260929 00:56:50.590261 2624 tablet_bootstrap.cc:654] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:50.590646 2625 tablet_bootstrap.cc:654] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:50.591377 2624 log.cc:824] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:50.591706 2626 tablet_bootstrap.cc:654] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:50.591905 2625 log.cc:824] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:50.592474 2626 log.cc:824] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:50.593353 2624 tablet_bootstrap.cc:492] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: No bootstrap required, opened a new log
I20260929 00:56:50.593530 2624 ts_tablet_manager.cc:1397] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 00:56:50.593830 2625 tablet_bootstrap.cc:492] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: No bootstrap required, opened a new log
I20260929 00:56:50.594022 2625 ts_tablet_manager.cc:1397] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:56:50.594273 2626 tablet_bootstrap.cc:492] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: No bootstrap required, opened a new log
I20260929 00:56:50.594422 2626 ts_tablet_manager.cc:1397] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 00:56:50.596736 2625 raft_consensus.cc:348] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.596956 2625 raft_consensus.cc:374] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:50.597003 2625 raft_consensus.cc:729] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b706f745217546d3be5b4da860a15bc4, State: Initialized, Role: FOLLOWER
I20260929 00:56:50.597432 2625 consensus_queue.cc:260] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.597975 2321 heartbeater.cc:503] Master 127.0.98.62:35797 was elected leader, sending a full tablet report...
I20260929 00:56:50.598026 2624 raft_consensus.cc:348] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.598304 2624 raft_consensus.cc:374] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:50.598362 2624 raft_consensus.cc:729] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daffb0824884f4190a8d26143ef33b6, State: Initialized, Role: FOLLOWER
I20260929 00:56:50.598410 2625 ts_tablet_manager.cc:1428] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260929 00:56:50.598755 2624 consensus_queue.cc:260] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.599252 2626 raft_consensus.cc:348] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.599468 2626 raft_consensus.cc:374] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:50.599483 2461 heartbeater.cc:503] Master 127.0.98.62:35797 was elected leader, sending a full tablet report...
I20260929 00:56:50.599512 2626 raft_consensus.cc:729] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c206d894043048c9b85dfec92de3eb6a, State: Initialized, Role: FOLLOWER
I20260929 00:56:50.599673 2624 ts_tablet_manager.cc:1428] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 00:56:50.599921 2626 consensus_queue.cc:260] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.600450 2601 heartbeater.cc:503] Master 127.0.98.62:35797 was elected leader, sending a full tablet report...
I20260929 00:56:50.600780 2626 ts_tablet_manager.cc:1428] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:56:50.668277 2631 raft_consensus.cc:482] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:50.668471 2631 raft_consensus.cc:504] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.669620 2631 leader_election.cc:288] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615), b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635)
I20260929 00:56:50.673936 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.670032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37765 (local address 127.0.98.3:35615)
0929 00:56:50.670219 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.670224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.670246 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:56:50.670308 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.670314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.670367 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.670450 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:50.670732 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.671602 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.672877 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.673470 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.673551 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.673603 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.673791 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.673845 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:56:50.673964 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.669833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35615 (local address 127.0.98.2:37765)
0929 00:56:50.670029 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.670056 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.670152 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.670469 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.670471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.670488 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:50.670620 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.670627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.671697 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.671700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.672763 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.672771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.673574 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.673578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.673676 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.673699 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.673770 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.673835 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":33}
I20260929 00:56:50.674682 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.669920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55219 (local address 127.0.98.1:41635)
0929 00:56:50.670056 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.670061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.670561 (+ 500us) server_negotiation.cc:407] Connection header received
0929 00:56:50.670670 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.670675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.670735 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.670797 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:50.671526 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.672255 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.673452 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.674009 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.674106 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.674169 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.674469 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.674538 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:56:50.674681 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840c20622b9243c39b1b63b21dd20090" candidate_uuid: "0daffb0824884f4190a8d26143ef33b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c206d894043048c9b85dfec92de3eb6a" is_pre_election: true
I20260929 00:56:50.674973 2555 raft_consensus.cc:2454] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0daffb0824884f4190a8d26143ef33b6 in term 0.
I20260929 00:56:50.675050 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.669833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41635 (local address 127.0.98.2:55219)
0929 00:56:50.670465 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.670478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.670575 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.670851 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.670853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.670870 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:50.671413 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.671422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.672360 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.672362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.673336 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.673345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.674223 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.674227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.674315 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.674341 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.674831 (+ 490us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.674895 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":265,"thread_start_us":153,"threads_started":1}
I20260929 00:56:50.675457 2346 leader_election.cc:302] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daffb0824884f4190a8d26143ef33b6, c206d894043048c9b85dfec92de3eb6a; no voters:
I20260929 00:56:50.675668 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840c20622b9243c39b1b63b21dd20090" candidate_uuid: "0daffb0824884f4190a8d26143ef33b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706f745217546d3be5b4da860a15bc4" is_pre_election: true
I20260929 00:56:50.675740 2631 raft_consensus.cc:2781] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:50.675805 2631 raft_consensus.cc:482] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:50.675872 2631 raft_consensus.cc:3037] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:50.675956 2275 raft_consensus.cc:2454] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0daffb0824884f4190a8d26143ef33b6 in term 0.
I20260929 00:56:50.676915 2631 raft_consensus.cc:504] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.677294 2631 leader_election.cc:288] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [CANDIDATE]: Term 1 election: Requested vote from peers c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615), b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635)
I20260929 00:56:50.677585 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840c20622b9243c39b1b63b21dd20090" candidate_uuid: "0daffb0824884f4190a8d26143ef33b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c206d894043048c9b85dfec92de3eb6a"
I20260929 00:56:50.677660 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840c20622b9243c39b1b63b21dd20090" candidate_uuid: "0daffb0824884f4190a8d26143ef33b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706f745217546d3be5b4da860a15bc4"
I20260929 00:56:50.677696 2555 raft_consensus.cc:3037] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:50.677800 2275 raft_consensus.cc:3037] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:50.678763 2555 raft_consensus.cc:2454] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0daffb0824884f4190a8d26143ef33b6 in term 1.
I20260929 00:56:50.678916 2275 raft_consensus.cc:2454] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0daffb0824884f4190a8d26143ef33b6 in term 1.
I20260929 00:56:50.679075 2346 leader_election.cc:302] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daffb0824884f4190a8d26143ef33b6, c206d894043048c9b85dfec92de3eb6a; no voters:
I20260929 00:56:50.679255 2631 raft_consensus.cc:2781] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:50.679499 2631 raft_consensus.cc:686] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0daffb0824884f4190a8d26143ef33b6, State: Running, Role: LEADER
I20260929 00:56:50.679827 2631 consensus_queue.cc:237] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } }
I20260929 00:56:50.682780 2118 catalog_manager.cc:5495] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0daffb0824884f4190a8d26143ef33b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:56:50 dist-test-slave-30s6 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:50.735944 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.727759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.0.1:41356)
0929 00:56:50.728046 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.728058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.728196 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.728429 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.728431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.728888 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:50.728974 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.728980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.729704 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.729709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.730330 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.730338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.730747 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.730749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.730825 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.731428 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:50.731457 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:50.732800 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:50.734274 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.734284 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.734290 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.734614 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.734893 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:50.734897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:50.734899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:50.734978 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:50.735332 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:50.735340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:50.735444 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.735550 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.735697 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
I20260929 00:56:50.735944 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.727842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.0.98.62:35797)
0929 00:56:50.727973 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.727978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.728133 (+ 155us) server_negotiation.cc:407] Connection header received
0929 00:56:50.728288 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.728293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.728334 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.728412 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:50.729089 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.729594 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.730438 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.730644 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.730745 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.732935 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:50.732938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:50.732940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:50.732957 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:50.734123 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.734733 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.734737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.734739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.734810 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:50.735065 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:50.735068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:50.735069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:50.735227 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:50.735323 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.735604 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.735748 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":48}
I20260929 00:56:50.739641 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41356:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:56:50.740312 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:50.744719 2255 tablet_service.cc:1467] Processing CreateTablet for tablet a7ebc7ae3d824ef7800adecde21c4d7f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28c7d16a70c143089cbd7a9caf4a9419]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:56:50.744804 2395 tablet_service.cc:1467] Processing CreateTablet for tablet a7ebc7ae3d824ef7800adecde21c4d7f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28c7d16a70c143089cbd7a9caf4a9419]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:56:50.744951 2535 tablet_service.cc:1467] Processing CreateTablet for tablet a7ebc7ae3d824ef7800adecde21c4d7f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28c7d16a70c143089cbd7a9caf4a9419]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:56:50.745386 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ebc7ae3d824ef7800adecde21c4d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:50.745460 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ebc7ae3d824ef7800adecde21c4d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:50.745460 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ebc7ae3d824ef7800adecde21c4d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:50.747807 2625 tablet_bootstrap.cc:492] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: Bootstrap starting.
I20260929 00:56:50.748188 2626 tablet_bootstrap.cc:492] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: Bootstrap starting.
I20260929 00:56:50.748929 2625 tablet_bootstrap.cc:654] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:50.749183 2626 tablet_bootstrap.cc:654] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:50.750079 2624 tablet_bootstrap.cc:492] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: Bootstrap starting.
I20260929 00:56:50.750370 2625 tablet_bootstrap.cc:492] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: No bootstrap required, opened a new log
I20260929 00:56:50.750463 2625 ts_tablet_manager.cc:1397] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:56:50.750847 2624 tablet_bootstrap.cc:654] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:50.750842 2625 raft_consensus.cc:348] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.750970 2625 raft_consensus.cc:374] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:50.751003 2625 raft_consensus.cc:729] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b706f745217546d3be5b4da860a15bc4, State: Initialized, Role: FOLLOWER
I20260929 00:56:50.751125 2625 consensus_queue.cc:260] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.751415 2625 ts_tablet_manager.cc:1428] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:56:50.751387 2630 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.751528 2630 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:50.751748 2626 tablet_bootstrap.cc:492] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: No bootstrap required, opened a new log
I20260929 00:56:50.751828 2626 ts_tablet_manager.cc:1397] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 00:56:50.752265 2626 raft_consensus.cc:348] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.752369 2626 raft_consensus.cc:374] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:50.752408 2626 raft_consensus.cc:729] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c206d894043048c9b85dfec92de3eb6a, State: Initialized, Role: FOLLOWER
I20260929 00:56:50.752542 2626 consensus_queue.cc:260] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.752831 2632 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.752957 2632 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:50.752846 2626 ts_tablet_manager.cc:1428] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:56:50.753628 2624 tablet_bootstrap.cc:492] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: No bootstrap required, opened a new log
I20260929 00:56:50.753729 2624 ts_tablet_manager.cc:1397] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 00:56:50.754159 2624 raft_consensus.cc:348] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.754302 2624 raft_consensus.cc:374] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:50.754326 2624 raft_consensus.cc:729] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daffb0824884f4190a8d26143ef33b6, State: Initialized, Role: FOLLOWER
I20260929 00:56:50.754458 2624 consensus_queue.cc:260] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.754733 2624 ts_tablet_manager.cc:1428] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 00:56:50.754698 2631 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.754884 2631 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 00:56:50.769541 2602 tablet.cc:2369] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:56:50.789989 2462 tablet.cc:2369] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:56:50.822376 2322 tablet.cc:2369] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:56:50.854321 2632 raft_consensus.cc:482] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:50.854444 2632 raft_consensus.cc:504] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.855576 2632 leader_election.cc:288] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635), 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945)
I20260929 00:56:50.859011 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.855737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41635 (local address 127.0.98.3:60629)
0929 00:56:50.855883 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.855899 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.855990 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.856239 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.856242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.856261 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:50.856374 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.856379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.857094 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.857100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.857909 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.857917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.858594 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.858597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.858695 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.858712 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.858802 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.858876 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:56:50.859004 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.855867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60629 (local address 127.0.98.1:41635)
0929 00:56:50.855998 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.856003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.856019 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:56:50.856070 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.856076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.856131 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.856226 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:50.856488 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.856983 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.858031 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.858466 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.858585 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.858647 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.858815 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.858886 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:56:50.859383 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.855990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36599 (local address 127.0.98.2:32945)
0929 00:56:50.856150 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.856155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.856312 (+ 157us) server_negotiation.cc:407] Connection header received
0929 00:56:50.856461 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.856464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.856506 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.856586 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:50.856849 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.857325 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.858387 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.858783 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.858904 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.858962 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.859203 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.859274 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 00:56:50.859462 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ebc7ae3d824ef7800adecde21c4d7f" candidate_uuid: "c206d894043048c9b85dfec92de3eb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706f745217546d3be5b4da860a15bc4" is_pre_election: true
I20260929 00:56:50.859620 2275 raft_consensus.cc:2454] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c206d894043048c9b85dfec92de3eb6a in term 0.
I20260929 00:56:50.859752 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.855816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.98.3:36599)
0929 00:56:50.856224 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.856235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.856345 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.856595 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.856598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.856612 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:50.856744 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.856749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.857427 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.857433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.858268 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.858276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.858916 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.858920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.859014 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.859035 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.859521 (+ 486us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.859592 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":124,"threads_started":1}
I20260929 00:56:50.859966 2487 leader_election.cc:302] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b706f745217546d3be5b4da860a15bc4, c206d894043048c9b85dfec92de3eb6a; no voters:
I20260929 00:56:50.860253 2632 raft_consensus.cc:2781] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:50.860318 2632 raft_consensus.cc:482] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:50.860354 2632 raft_consensus.cc:3037] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:50.860378 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ebc7ae3d824ef7800adecde21c4d7f" candidate_uuid: "c206d894043048c9b85dfec92de3eb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daffb0824884f4190a8d26143ef33b6" is_pre_election: true
I20260929 00:56:50.860587 2415 raft_consensus.cc:2454] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c206d894043048c9b85dfec92de3eb6a in term 0.
I20260929 00:56:50.861485 2632 raft_consensus.cc:504] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.861783 2632 leader_election.cc:288] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [CANDIDATE]: Term 1 election: Requested vote from peers b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635), 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945)
I20260929 00:56:50.862035 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ebc7ae3d824ef7800adecde21c4d7f" candidate_uuid: "c206d894043048c9b85dfec92de3eb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706f745217546d3be5b4da860a15bc4"
I20260929 00:56:50.862116 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ebc7ae3d824ef7800adecde21c4d7f" candidate_uuid: "c206d894043048c9b85dfec92de3eb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daffb0824884f4190a8d26143ef33b6"
I20260929 00:56:50.862182 2275 raft_consensus.cc:3037] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:50.862250 2415 raft_consensus.cc:3037] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:50.863052 2275 raft_consensus.cc:2454] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c206d894043048c9b85dfec92de3eb6a in term 1.
I20260929 00:56:50.863092 2415 raft_consensus.cc:2454] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c206d894043048c9b85dfec92de3eb6a in term 1.
I20260929 00:56:50.863307 2487 leader_election.cc:302] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b706f745217546d3be5b4da860a15bc4, c206d894043048c9b85dfec92de3eb6a; no voters:
I20260929 00:56:50.863476 2632 raft_consensus.cc:2781] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:50.863677 2632 raft_consensus.cc:686] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [term 1 LEADER]: Becoming Leader. State: Replica: c206d894043048c9b85dfec92de3eb6a, State: Running, Role: LEADER
I20260929 00:56:50.863981 2632 consensus_queue.cc:237] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:50.866222 2646 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: TxnStatusTablet state changed. Reason: New leader c206d894043048c9b85dfec92de3eb6a. Latest consensus state: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.866321 2646 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:56:50.866585 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:56:50.866794 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:56:50.867089 2118 catalog_manager.cc:5495] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c206d894043048c9b85dfec92de3eb6a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } health_report { overall_health: HEALTHY } } }
I20260929 00:56:50.908000 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.904036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35615 (local address 127.0.0.1:41408)
0929 00:56:50.904203 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.904229 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.904325 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.904677 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.904679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.904685 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:56:50.904769 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.904773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.905760 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.905762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.906405 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.906414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.906963 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.906965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.907026 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.907066 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:56:50.907773 (+ 707us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:56:50.907783 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.907822 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.907884 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 00:56:50.908020 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.904150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.0.98.3:35615)
0929 00:56:50.904275 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.904280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.904307 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:56:50.904492 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.904496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.904560 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.904664 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:50.904871 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.905663 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.906514 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.906865 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.906961 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.907192 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:50.907689 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:50.907796 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.907832 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.907883 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:56:50.912847 2415 raft_consensus.cc:1261] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Refusing update from remote peer c206d894043048c9b85dfec92de3eb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:50.912851 2275 raft_consensus.cc:1261] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Refusing update from remote peer c206d894043048c9b85dfec92de3eb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:50.913507 2632 consensus_queue.cc:1035] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:50.913698 2646 consensus_queue.cc:1035] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:50.916671 2631 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: TxnStatusTablet state changed. Reason: New leader c206d894043048c9b85dfec92de3eb6a. Latest consensus state: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.916781 2631 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:50.917888 2630 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: TxnStatusTablet state changed. Reason: New leader c206d894043048c9b85dfec92de3eb6a. Latest consensus state: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.917999 2630 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:50.919454 2632 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.919556 2632 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:56:50.920523 2631 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.920625 2631 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P 0daffb0824884f4190a8d26143ef33b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:50.921188 2646 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.921284 2646 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P c206d894043048c9b85dfec92de3eb6a: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:56:50.921411 2652 mvcc.cc:205] Tried to move back new op lower bound from 7334475411092606976 to 7334475410901475328. Current Snapshot: MvccSnapshot[applied={T|T < 7334475411092606976}]
I20260929 00:56:50.922262 2630 tablet_replica.cc:439] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } } }
I20260929 00:56:50.922384 2630 tablet_replica.cc:442] T a7ebc7ae3d824ef7800adecde21c4d7f P b706f745217546d3be5b4da860a15bc4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:56:50.924587 2653 mvcc.cc:205] Tried to move back new op lower bound from 7334475411092606976 to 7334475410901475328. Current Snapshot: MvccSnapshot[applied={T|T < 7334475411092606976}]
I20260929 00:56:50.930799 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.927342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.0.98.2:32945)
0929 00:56:50.927464 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.927470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.927504 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:56:50.927698 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.927700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.927751 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.927847 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:50.928086 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.928574 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.929424 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.929642 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.929748 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.930000 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:50.930429 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:50.930543 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.930614 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.930671 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:56:50.940609 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.937176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35615 (local address 127.0.98.2:50291)
0929 00:56:50.937530 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.937542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.937624 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.937847 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.937848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.937853 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:56:50.937945 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.937949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.938901 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.938903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.939489 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.939496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.940024 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.940026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.940098 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.940132 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:56:50.940397 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:56:50.940401 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.940446 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.940501 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260929 00:56:50.940618 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.937251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50291 (local address 127.0.98.3:35615)
0929 00:56:50.937368 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.937372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.937615 (+ 243us) server_negotiation.cc:407] Connection header received
0929 00:56:50.937716 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.937719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.937769 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.937845 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:50.938045 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.938804 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.939590 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.939916 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.940024 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.940256 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:50.940323 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:50.940390 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.940456 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.940501 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:56:50.947067 2275 raft_consensus.cc:1261] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Refusing update from remote peer 0daffb0824884f4190a8d26143ef33b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:50.947257 2555 raft_consensus.cc:1261] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Refusing update from remote peer 0daffb0824884f4190a8d26143ef33b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:50.947554 2634 consensus_queue.cc:1035] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:50.947715 2631 consensus_queue.cc:1035] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:50.968925 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.964854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.98.3:60881)
0929 00:56:50.965019 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:50.965032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:50.965097 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:50.965572 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:50.965578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:50.965588 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:56:50.965700 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.965706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.966758 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.966762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.967726 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:50.967736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.968181 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.968185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:50.968277 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.968325 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:56:50.968634 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:56:50.968640 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:56:50.968719 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 00:56:50.968790 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 00:56:50.968890 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:50.965204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60881 (local address 127.0.98.2:32945)
0929 00:56:50.965359 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:50.965366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:50.965382 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:56:50.965415 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:50.965417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:50.965458 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:50.965514 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:50.965829 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.966628 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.967859 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:50.968076 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:50.968160 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:50.968474 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:50.968543 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:50.968611 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:50.968736 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 00:56:50.968782 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 00:56:50.990618 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:50.990803 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:50.993021 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: Sending DeleteTablet for 3 replicas of tablet 840c20622b9243c39b1b63b21dd20090
I20260929 00:56:50.993785 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 840c20622b9243c39b1b63b21dd20090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41394
I20260929 00:56:50.994316 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 840c20622b9243c39b1b63b21dd20090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260929 00:56:50.994787 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 840c20622b9243c39b1b63b21dd20090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35690
I20260929 00:56:50.994856 2670 tablet_replica.cc:330] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: stopping tablet replica
I20260929 00:56:50.995143 2670 raft_consensus.cc:2229] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:50.995083 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:56:50.995298 2670 raft_consensus.cc:2258] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 00:56:50.995502 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:50.996799 2670 ts_tablet_manager.cc:1905] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:50.999918 2670 ts_tablet_manager.cc:1918] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:51.000010 2670 log.cc:1196] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/840c20622b9243c39b1b63b21dd20090
I20260929 00:56:51.000378 2670 ts_tablet_manager.cc:1939] T 840c20622b9243c39b1b63b21dd20090 P c206d894043048c9b85dfec92de3eb6a: Deleting consensus metadata
I20260929 00:56:51.000942 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8ceecb6ed44e1e89452087291f3e8b (DEFAULT_TABLE table=test-table [id=36cbdb1bd832493ea02e4554c5831571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:51.001201 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8ceecb6ed44e1e89452087291f3e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:51.002460 2102 catalog_manager.cc:4841] TS c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615): tablet 840c20622b9243c39b1b63b21dd20090 (table test-table [id=482bd3dd9e9f45eebcfa2dcc37f417b0]) successfully deleted
I20260929 00:56:51.002939 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8ceecb6ed44e1e89452087291f3e8b (DEFAULT_TABLE table=test-table [id=36cbdb1bd832493ea02e4554c5831571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:51.003175 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8ceecb6ed44e1e89452087291f3e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:51.003573 2624 tablet_bootstrap.cc:492] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: Bootstrap starting.
I20260929 00:56:51.004496 2624 tablet_bootstrap.cc:654] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:51.005301 2626 tablet_bootstrap.cc:492] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: Bootstrap starting.
I20260929 00:56:51.005999 2671 tablet_replica.cc:330] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: stopping tablet replica
I20260929 00:56:51.006245 2626 tablet_bootstrap.cc:654] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:51.006402 2671 raft_consensus.cc:2229] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:51.006716 2671 raft_consensus.cc:2258] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:51.006399 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8ceecb6ed44e1e89452087291f3e8b (DEFAULT_TABLE table=test-table [id=36cbdb1bd832493ea02e4554c5831571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:51.006891 2672 tablet_replica.cc:330] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: stopping tablet replica
I20260929 00:56:51.007047 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8ceecb6ed44e1e89452087291f3e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:51.007179 2672 raft_consensus.cc:2229] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:56:51.007510 2672 raft_consensus.cc:2258] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:51.008847 2672 ts_tablet_manager.cc:1905] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:51.009016 2625 tablet_bootstrap.cc:492] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: Bootstrap starting.
I20260929 00:56:51.009935 2625 tablet_bootstrap.cc:654] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:51.010957 2625 tablet_bootstrap.cc:492] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: No bootstrap required, opened a new log
I20260929 00:56:51.011034 2625 ts_tablet_manager.cc:1397] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 00:56:51.011478 2625 raft_consensus.cc:348] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.011618 2625 raft_consensus.cc:374] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:51.011621 2672 ts_tablet_manager.cc:1918] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:51.011660 2625 raft_consensus.cc:729] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b706f745217546d3be5b4da860a15bc4, State: Initialized, Role: FOLLOWER
I20260929 00:56:51.011704 2672 log.cc:1196] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/840c20622b9243c39b1b63b21dd20090
I20260929 00:56:51.011783 2625 consensus_queue.cc:260] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.012041 2672 ts_tablet_manager.cc:1939] T 840c20622b9243c39b1b63b21dd20090 P 0daffb0824884f4190a8d26143ef33b6: Deleting consensus metadata
I20260929 00:56:51.012048 2625 ts_tablet_manager.cc:1428] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 00:56:51.012729 2671 ts_tablet_manager.cc:1905] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:51.013087 2104 catalog_manager.cc:4841] TS 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945): tablet 840c20622b9243c39b1b63b21dd20090 (table test-table [id=482bd3dd9e9f45eebcfa2dcc37f417b0]) successfully deleted
I20260929 00:56:51.014839 2626 tablet_bootstrap.cc:492] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: No bootstrap required, opened a new log
I20260929 00:56:51.014838 2624 tablet_bootstrap.cc:492] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: No bootstrap required, opened a new log
I20260929 00:56:51.014925 2626 ts_tablet_manager.cc:1397] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260929 00:56:51.014925 2624 ts_tablet_manager.cc:1397] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260929 00:56:51.015341 2624 raft_consensus.cc:348] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.015341 2626 raft_consensus.cc:348] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.015452 2624 raft_consensus.cc:374] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:51.015452 2626 raft_consensus.cc:374] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:51.015487 2624 raft_consensus.cc:729] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daffb0824884f4190a8d26143ef33b6, State: Initialized, Role: FOLLOWER
I20260929 00:56:51.015497 2626 raft_consensus.cc:729] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c206d894043048c9b85dfec92de3eb6a, State: Initialized, Role: FOLLOWER
I20260929 00:56:51.015611 2624 consensus_queue.cc:260] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.015614 2626 consensus_queue.cc:260] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.015877 2624 ts_tablet_manager.cc:1428] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:51.015870 2626 ts_tablet_manager.cc:1428] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:56:51.020706 2671 ts_tablet_manager.cc:1918] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:51.020807 2671 log.cc:1196] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/840c20622b9243c39b1b63b21dd20090
I20260929 00:56:51.021167 2671 ts_tablet_manager.cc:1939] T 840c20622b9243c39b1b63b21dd20090 P b706f745217546d3be5b4da860a15bc4: Deleting consensus metadata
I20260929 00:56:51.022675 2103 catalog_manager.cc:4841] TS b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635): tablet 840c20622b9243c39b1b63b21dd20090 (table test-table [id=482bd3dd9e9f45eebcfa2dcc37f417b0]) successfully deleted
I20260929 00:56:51.224432 2634 raft_consensus.cc:482] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:51.224573 2634 raft_consensus.cc:504] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.225098 2634 leader_election.cc:288] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635), c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615)
I20260929 00:56:51.225406 2630 raft_consensus.cc:482] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:51.225539 2630 raft_consensus.cc:504] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.225608 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8ceecb6ed44e1e89452087291f3e8b" candidate_uuid: "0daffb0824884f4190a8d26143ef33b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c206d894043048c9b85dfec92de3eb6a" is_pre_election: true
I20260929 00:56:51.225610 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8ceecb6ed44e1e89452087291f3e8b" candidate_uuid: "0daffb0824884f4190a8d26143ef33b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706f745217546d3be5b4da860a15bc4" is_pre_election: true
I20260929 00:56:51.225728 2555 raft_consensus.cc:2454] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0daffb0824884f4190a8d26143ef33b6 in term 0.
I20260929 00:56:51.226023 2275 raft_consensus.cc:2454] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0daffb0824884f4190a8d26143ef33b6 in term 0.
I20260929 00:56:51.226133 2346 leader_election.cc:302] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daffb0824884f4190a8d26143ef33b6, c206d894043048c9b85dfec92de3eb6a; no voters:
I20260929 00:56:51.226344 2634 raft_consensus.cc:2781] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:51.226406 2634 raft_consensus.cc:482] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:51.226449 2634 raft_consensus.cc:3037] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:51.226825 2630 leader_election.cc:288] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945), c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615)
I20260929 00:56:51.227833 2634 raft_consensus.cc:504] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.228196 2634 leader_election.cc:288] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [CANDIDATE]: Term 1 election: Requested vote from peers b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635), c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615)
I20260929 00:56:51.228494 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8ceecb6ed44e1e89452087291f3e8b" candidate_uuid: "0daffb0824884f4190a8d26143ef33b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706f745217546d3be5b4da860a15bc4"
I20260929 00:56:51.228595 2275 raft_consensus.cc:3037] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:51.228561 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8ceecb6ed44e1e89452087291f3e8b" candidate_uuid: "0daffb0824884f4190a8d26143ef33b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c206d894043048c9b85dfec92de3eb6a"
I20260929 00:56:51.228691 2555 raft_consensus.cc:3037] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:51.229694 2275 raft_consensus.cc:2454] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0daffb0824884f4190a8d26143ef33b6 in term 1.
I20260929 00:56:51.229818 2555 raft_consensus.cc:2454] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0daffb0824884f4190a8d26143ef33b6 in term 1.
I20260929 00:56:51.230072 2347 leader_election.cc:302] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daffb0824884f4190a8d26143ef33b6, b706f745217546d3be5b4da860a15bc4; no voters:
I20260929 00:56:51.230239 2634 raft_consensus.cc:2781] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:51.230332 2634 raft_consensus.cc:686] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0daffb0824884f4190a8d26143ef33b6, State: Running, Role: LEADER
I20260929 00:56:51.230475 2634 consensus_queue.cc:237] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:51.231772 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:51.227364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53451 (local address 127.0.98.2:32945)
0929 00:56:51.227504 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:51.227509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:51.227524 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:56:51.227567 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:51.227570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:51.227615 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:51.227694 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:51.228068 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:51.228822 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:51.230395 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:51.230971 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:51.231032 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:51.231083 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:51.231592 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 00:56:51.231666 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:56:51.232216 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:51.227021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.98.1:53451)
0929 00:56:51.227365 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:51.227381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:51.227482 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:51.227759 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:51.227762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:51.227777 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:51.227881 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:51.227888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:51.228949 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:51.228953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:51.230255 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:51.230268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:51.231175 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:51.231177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:51.231455 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:51.231487 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:56:51.231579 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 00:56:51.231649 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":194,"thread_start_us":86,"threads_started":1}
I20260929 00:56:51.232237 2117 catalog_manager.cc:5495] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0daffb0824884f4190a8d26143ef33b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } health_report { overall_health: UNKNOWN } } }
I20260929 00:56:51.232576 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:51.227661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50645 (local address 127.0.98.3:35615)
0929 00:56:51.227776 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:51.227780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:51.227791 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:56:51.227832 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:51.227835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:51.227873 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:51.227950 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:51.228242 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:51.228942 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:51.230467 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:51.230981 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:51.231061 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:51.231147 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:51.232400 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0929 00:56:51.232469 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:56:51.232577 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8ceecb6ed44e1e89452087291f3e8b" candidate_uuid: "b706f745217546d3be5b4da860a15bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daffb0824884f4190a8d26143ef33b6" is_pre_election: true
I20260929 00:56:51.232928 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:51.227052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35615 (local address 127.0.98.1:50645)
0929 00:56:51.227577 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:51.227591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:51.227660 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:51.227980 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:51.227983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:51.228002 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:51.228088 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:51.228105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:51.229094 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:51.229109 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:51.230331 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:51.230343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:51.231241 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:51.231244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:51.231352 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:51.231373 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:56:51.232692 (+ 1319us) client_negotiation.cc:240] Negotiation successful
0929 00:56:51.232771 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":29,"thread_start_us":116,"threads_started":1}
I20260929 00:56:51.233358 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8ceecb6ed44e1e89452087291f3e8b" candidate_uuid: "b706f745217546d3be5b4da860a15bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c206d894043048c9b85dfec92de3eb6a" is_pre_election: true
I20260929 00:56:51.233533 2555 raft_consensus.cc:2379] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b706f745217546d3be5b4da860a15bc4 in current term 1: Already voted for candidate 0daffb0824884f4190a8d26143ef33b6 in this term.
I20260929 00:56:51.233837 2206 leader_election.cc:302] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b706f745217546d3be5b4da860a15bc4; no voters: 0daffb0824884f4190a8d26143ef33b6, c206d894043048c9b85dfec92de3eb6a
I20260929 00:56:51.234072 2630 raft_consensus.cc:2726] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 00:56:51.259222 2555 raft_consensus.cc:1261] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Refusing update from remote peer 0daffb0824884f4190a8d26143ef33b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:51.259320 2275 raft_consensus.cc:1261] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Refusing update from remote peer 0daffb0824884f4190a8d26143ef33b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:51.259617 2634 consensus_queue.cc:1035] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:51.259774 2631 consensus_queue.cc:1035] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:51.263830 2651 mvcc.cc:205] Tried to move back new op lower bound from 7334475412514459648 to 7334475412401418240. Current Snapshot: MvccSnapshot[applied={T|T < 7334475412514459648}]
W20260929 00:56:51.273402 2652 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 00:56:52.826292 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:52.826432 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:52.828639 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: Sending DeleteTablet for 3 replicas of tablet 4d8ceecb6ed44e1e89452087291f3e8b
I20260929 00:56:52.829156 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8ceecb6ed44e1e89452087291f3e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260929 00:56:52.829175 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8ceecb6ed44e1e89452087291f3e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35690
I20260929 00:56:52.829309 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8ceecb6ed44e1e89452087291f3e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41394
I20260929 00:56:52.829488 2702 tablet_replica.cc:330] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: stopping tablet replica
I20260929 00:56:52.829617 2702 raft_consensus.cc:2229] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:52.829727 2702 raft_consensus.cc:2258] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:52.829946 2703 tablet_replica.cc:330] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: stopping tablet replica
I20260929 00:56:52.830101 2703 raft_consensus.cc:2229] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:56:52.830318 2703 raft_consensus.cc:2258] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:52.830488 2702 ts_tablet_manager.cc:1905] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:52.830916 2703 ts_tablet_manager.cc:1905] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:52.832526 2704 tablet_replica.cc:330] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: stopping tablet replica
I20260929 00:56:52.832558 2702 ts_tablet_manager.cc:1918] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:56:52.832645 2704 raft_consensus.cc:2229] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:52.832687 2702 log.cc:1196] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/4d8ceecb6ed44e1e89452087291f3e8b
I20260929 00:56:52.832757 2704 raft_consensus.cc:2258] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:52.833045 2702 ts_tablet_manager.cc:1939] T 4d8ceecb6ed44e1e89452087291f3e8b P b706f745217546d3be5b4da860a15bc4: Deleting consensus metadata
I20260929 00:56:52.833302 2704 ts_tablet_manager.cc:1905] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:52.833716 2103 catalog_manager.cc:4841] TS b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635): tablet 4d8ceecb6ed44e1e89452087291f3e8b (table test-table [id=36cbdb1bd832493ea02e4554c5831571]) successfully deleted
I20260929 00:56:52.833760 2703 ts_tablet_manager.cc:1918] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:56:52.833824 2703 log.cc:1196] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/4d8ceecb6ed44e1e89452087291f3e8b
I20260929 00:56:52.834102 2703 ts_tablet_manager.cc:1939] T 4d8ceecb6ed44e1e89452087291f3e8b P 0daffb0824884f4190a8d26143ef33b6: Deleting consensus metadata
I20260929 00:56:52.834802 2104 catalog_manager.cc:4841] TS 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945): tablet 4d8ceecb6ed44e1e89452087291f3e8b (table test-table [id=36cbdb1bd832493ea02e4554c5831571]) successfully deleted
I20260929 00:56:52.836337 2704 ts_tablet_manager.cc:1918] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:56:52.836419 2704 log.cc:1196] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/4d8ceecb6ed44e1e89452087291f3e8b
I20260929 00:56:52.836679 2704 ts_tablet_manager.cc:1939] T 4d8ceecb6ed44e1e89452087291f3e8b P c206d894043048c9b85dfec92de3eb6a: Deleting consensus metadata
I20260929 00:56:52.837334 2102 catalog_manager.cc:4841] TS c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615): tablet 4d8ceecb6ed44e1e89452087291f3e8b (table test-table [id=36cbdb1bd832493ea02e4554c5831571]) successfully deleted
Sep 29 00:56:52 dist-test-slave-30s6 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:56:52 dist-test-slave-30s6 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:52.858407 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:52.849964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.0.98.62:35797)
0929 00:56:52.850234 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:52.850242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:52.850341 (+ 99us) server_negotiation.cc:407] Connection header received
0929 00:56:52.850529 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:52.850532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:52.850585 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:52.850683 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:52.851360 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:52.851907 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:52.852776 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:52.853011 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:52.853119 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:52.855439 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:52.855443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:52.855445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:52.855468 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:52.856739 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:52.857141 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:52.857146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:52.857148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:52.857229 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:52.857509 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:52.857514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:52.857515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:52.857689 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:52.857827 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:52.858050 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 00:56:52.858167 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260929 00:56:52.860675 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:56:52.861102 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:52.865542 2255 tablet_service.cc:1467] Processing CreateTablet for tablet db5a703efdf048c0802b177359d355a0 (DEFAULT_TABLE table=test-table [id=b52ff44eb70d46c8921cf8a7b03d5ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:52.865558 2395 tablet_service.cc:1467] Processing CreateTablet for tablet db5a703efdf048c0802b177359d355a0 (DEFAULT_TABLE table=test-table [id=b52ff44eb70d46c8921cf8a7b03d5ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:52.865828 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5a703efdf048c0802b177359d355a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:52.865828 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5a703efdf048c0802b177359d355a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:52.868449 2534 tablet_service.cc:1467] Processing CreateTablet for tablet db5a703efdf048c0802b177359d355a0 (DEFAULT_TABLE table=test-table [id=b52ff44eb70d46c8921cf8a7b03d5ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:52.868571 2714 tablet_bootstrap.cc:492] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: Bootstrap starting.
I20260929 00:56:52.868738 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5a703efdf048c0802b177359d355a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:52.869439 2714 tablet_bootstrap.cc:654] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:52.870208 2713 tablet_bootstrap.cc:492] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: Bootstrap starting.
I20260929 00:56:52.870816 2714 tablet_bootstrap.cc:492] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: No bootstrap required, opened a new log
I20260929 00:56:52.870973 2714 ts_tablet_manager.cc:1397] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:52.871032 2713 tablet_bootstrap.cc:654] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:52.871624 2714 raft_consensus.cc:348] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:52.871719 2714 raft_consensus.cc:374] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:52.871740 2714 raft_consensus.cc:729] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b706f745217546d3be5b4da860a15bc4, State: Initialized, Role: FOLLOWER
I20260929 00:56:52.871897 2714 consensus_queue.cc:260] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:52.871954 2716 tablet_bootstrap.cc:492] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: Bootstrap starting.
I20260929 00:56:52.872160 2713 tablet_bootstrap.cc:492] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: No bootstrap required, opened a new log
I20260929 00:56:52.872239 2713 ts_tablet_manager.cc:1397] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 00:56:52.872284 2714 ts_tablet_manager.cc:1428] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:52.872625 2713 raft_consensus.cc:348] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:52.872727 2716 tablet_bootstrap.cc:654] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:52.872757 2713 raft_consensus.cc:374] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:52.872798 2713 raft_consensus.cc:729] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daffb0824884f4190a8d26143ef33b6, State: Initialized, Role: FOLLOWER
I20260929 00:56:52.872957 2713 consensus_queue.cc:260] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:52.873251 2713 ts_tablet_manager.cc:1428] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:52.873911 2716 tablet_bootstrap.cc:492] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: No bootstrap required, opened a new log
I20260929 00:56:52.873996 2716 ts_tablet_manager.cc:1397] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 00:56:52.874449 2716 raft_consensus.cc:348] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:52.874555 2716 raft_consensus.cc:374] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:52.874590 2716 raft_consensus.cc:729] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c206d894043048c9b85dfec92de3eb6a, State: Initialized, Role: FOLLOWER
I20260929 00:56:52.874706 2716 consensus_queue.cc:260] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:52.874989 2716 ts_tablet_manager.cc:1428] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:52.997437 2646 raft_consensus.cc:482] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:52.997571 2646 raft_consensus.cc:504] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:52.998018 2646 leader_election.cc:288] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635), 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945)
I20260929 00:56:52.998479 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5a703efdf048c0802b177359d355a0" candidate_uuid: "c206d894043048c9b85dfec92de3eb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706f745217546d3be5b4da860a15bc4" is_pre_election: true
I20260929 00:56:52.998486 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5a703efdf048c0802b177359d355a0" candidate_uuid: "c206d894043048c9b85dfec92de3eb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daffb0824884f4190a8d26143ef33b6" is_pre_election: true
I20260929 00:56:52.998643 2415 raft_consensus.cc:2454] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c206d894043048c9b85dfec92de3eb6a in term 0.
I20260929 00:56:52.998641 2275 raft_consensus.cc:2454] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c206d894043048c9b85dfec92de3eb6a in term 0.
I20260929 00:56:52.999048 2487 leader_election.cc:302] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0daffb0824884f4190a8d26143ef33b6, b706f745217546d3be5b4da860a15bc4, c206d894043048c9b85dfec92de3eb6a; no voters:
I20260929 00:56:52.999271 2646 raft_consensus.cc:2781] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:52.999341 2646 raft_consensus.cc:482] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:52.999395 2646 raft_consensus.cc:3037] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:53.000370 2646 raft_consensus.cc:504] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.000687 2646 leader_election.cc:288] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [CANDIDATE]: Term 1 election: Requested vote from peers b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635), 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945)
I20260929 00:56:53.001024 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5a703efdf048c0802b177359d355a0" candidate_uuid: "c206d894043048c9b85dfec92de3eb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706f745217546d3be5b4da860a15bc4"
I20260929 00:56:53.001022 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5a703efdf048c0802b177359d355a0" candidate_uuid: "c206d894043048c9b85dfec92de3eb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daffb0824884f4190a8d26143ef33b6"
I20260929 00:56:53.001148 2415 raft_consensus.cc:3037] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:53.001149 2275 raft_consensus.cc:3037] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:53.002012 2275 raft_consensus.cc:2454] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c206d894043048c9b85dfec92de3eb6a in term 1.
I20260929 00:56:53.002177 2415 raft_consensus.cc:2454] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c206d894043048c9b85dfec92de3eb6a in term 1.
I20260929 00:56:53.002316 2487 leader_election.cc:302] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b706f745217546d3be5b4da860a15bc4, c206d894043048c9b85dfec92de3eb6a; no voters:
I20260929 00:56:53.002472 2646 raft_consensus.cc:2781] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:53.002599 2646 raft_consensus.cc:686] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 1 LEADER]: Becoming Leader. State: Replica: c206d894043048c9b85dfec92de3eb6a, State: Running, Role: LEADER
I20260929 00:56:53.002732 2646 consensus_queue.cc:237] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.004274 2118 catalog_manager.cc:5495] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c206d894043048c9b85dfec92de3eb6a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c206d894043048c9b85dfec92de3eb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } health_report { overall_health: HEALTHY } } }
I20260929 00:56:53.026296 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:53.023223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.0.98.3:35615)
0929 00:56:53.023471 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:53.023477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:53.023488 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:56:53.023521 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:53.023523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:53.023559 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:53.023624 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:53.023856 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.024381 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.025216 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.025454 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.025558 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:53.025796 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:53.025873 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:53.025949 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:53.026072 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 00:56:53.026122 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
I20260929 00:56:53.034142 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:53.031339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35615 (local address 127.0.98.3:57515)
0929 00:56:53.031633 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:53.031644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:53.031723 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:53.031907 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:53.031909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:53.031914 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:56:53.031994 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:53.031998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.032595 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.032597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:53.033213 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:53.033219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.033570 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.033572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:53.033625 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:53.033647 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:56:53.033890 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:56:53.033895 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:56:53.033953 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 00:56:53.034010 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":93,"threads_started":1}
I20260929 00:56:53.034149 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:53.031437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57515 (local address 127.0.98.3:35615)
0929 00:56:53.031543 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:53.031546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:53.031705 (+ 159us) server_negotiation.cc:407] Connection header received
0929 00:56:53.031794 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:53.031796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:53.031829 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:53.031886 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:53.032073 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.032516 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.033296 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.033485 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.033556 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:53.033758 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:53.033823 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:53.033874 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:53.033980 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 00:56:53.034016 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:56:53.039048 2415 raft_consensus.cc:1261] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Refusing update from remote peer c206d894043048c9b85dfec92de3eb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:53.039122 2275 raft_consensus.cc:1261] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Refusing update from remote peer c206d894043048c9b85dfec92de3eb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:53.039516 2646 consensus_queue.cc:1035] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:53.039703 2688 consensus_queue.cc:1035] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:53.069540 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:53.069696 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:53.071811 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: Sending DeleteTablet for 3 replicas of tablet db5a703efdf048c0802b177359d355a0
I20260929 00:56:53.072275 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet db5a703efdf048c0802b177359d355a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260929 00:56:53.072301 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet db5a703efdf048c0802b177359d355a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35690
I20260929 00:56:53.072434 2702 tablet_replica.cc:330] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: stopping tablet replica
I20260929 00:56:53.072525 2702 raft_consensus.cc:2229] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:53.072608 2702 raft_consensus.cc:2258] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:53.072736 2703 tablet_replica.cc:330] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: stopping tablet replica
I20260929 00:56:53.072825 2703 raft_consensus.cc:2229] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:53.072870 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet db5a703efdf048c0802b177359d355a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41394
I20260929 00:56:53.072904 2703 raft_consensus.cc:2258] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:53.073012 2704 tablet_replica.cc:330] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: stopping tablet replica
I20260929 00:56:53.073128 2704 raft_consensus.cc:2229] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:56:53.073365 2704 raft_consensus.cc:2258] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:53.073570 2703 ts_tablet_manager.cc:1905] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:53.073802 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:56:53.074016 2704 ts_tablet_manager.cc:1905] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 00:56:53.074191 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:53.076011 2703 ts_tablet_manager.cc:1918] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:53.076071 2703 log.cc:1196] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/db5a703efdf048c0802b177359d355a0
I20260929 00:56:53.076299 2703 ts_tablet_manager.cc:1939] T db5a703efdf048c0802b177359d355a0 P 0daffb0824884f4190a8d26143ef33b6: Deleting consensus metadata
I20260929 00:56:53.076460 2704 ts_tablet_manager.cc:1918] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:53.076535 2704 log.cc:1196] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/db5a703efdf048c0802b177359d355a0
I20260929 00:56:53.076773 2704 ts_tablet_manager.cc:1939] T db5a703efdf048c0802b177359d355a0 P c206d894043048c9b85dfec92de3eb6a: Deleting consensus metadata
I20260929 00:56:53.077512 2702 ts_tablet_manager.cc:1905] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:53.080639 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9521f40de341c096352935f63ff1ef (DEFAULT_TABLE table=test-table [id=dc61d975142048a892f1118aa5aed792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:53.080844 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9521f40de341c096352935f63ff1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:53.081104 2102 catalog_manager.cc:4841] TS c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615): tablet db5a703efdf048c0802b177359d355a0 (table test-table [id=b52ff44eb70d46c8921cf8a7b03d5ad0]) successfully deleted
I20260929 00:56:53.081364 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9521f40de341c096352935f63ff1ef (DEFAULT_TABLE table=test-table [id=dc61d975142048a892f1118aa5aed792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:53.081555 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9521f40de341c096352935f63ff1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:53.082791 2714 tablet_bootstrap.cc:492] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4: Bootstrap starting.
I20260929 00:56:53.083539 2716 tablet_bootstrap.cc:492] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: Bootstrap starting.
I20260929 00:56:53.083638 2714 tablet_bootstrap.cc:654] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:53.084184 2716 tablet_bootstrap.cc:654] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:53.085129 2702 ts_tablet_manager.cc:1918] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:56:53.085198 2702 log.cc:1196] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/db5a703efdf048c0802b177359d355a0
I20260929 00:56:53.085424 2702 ts_tablet_manager.cc:1939] T db5a703efdf048c0802b177359d355a0 P b706f745217546d3be5b4da860a15bc4: Deleting consensus metadata
I20260929 00:56:53.086017 2103 catalog_manager.cc:4841] TS b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635): tablet db5a703efdf048c0802b177359d355a0 (table test-table [id=b52ff44eb70d46c8921cf8a7b03d5ad0]) successfully deleted
I20260929 00:56:53.086282 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9521f40de341c096352935f63ff1ef (DEFAULT_TABLE table=test-table [id=dc61d975142048a892f1118aa5aed792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:53.086495 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9521f40de341c096352935f63ff1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:53.088383 2713 tablet_bootstrap.cc:492] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: Bootstrap starting.
I20260929 00:56:53.088394 2104 catalog_manager.cc:4841] TS 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945): tablet db5a703efdf048c0802b177359d355a0 (table test-table [id=b52ff44eb70d46c8921cf8a7b03d5ad0]) successfully deleted
I20260929 00:56:53.089128 2713 tablet_bootstrap.cc:654] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:53.089923 2716 tablet_bootstrap.cc:492] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: No bootstrap required, opened a new log
I20260929 00:56:53.090037 2716 ts_tablet_manager.cc:1397] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260929 00:56:53.090180 2713 tablet_bootstrap.cc:492] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: No bootstrap required, opened a new log
I20260929 00:56:53.090286 2713 ts_tablet_manager.cc:1397] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:53.090504 2716 raft_consensus.cc:348] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.090608 2716 raft_consensus.cc:374] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:53.090644 2716 raft_consensus.cc:729] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c206d894043048c9b85dfec92de3eb6a, State: Initialized, Role: FOLLOWER
I20260929 00:56:53.090613 2713 raft_consensus.cc:348] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.090713 2713 raft_consensus.cc:374] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:53.090754 2713 raft_consensus.cc:729] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daffb0824884f4190a8d26143ef33b6, State: Initialized, Role: FOLLOWER
I20260929 00:56:53.090755 2716 consensus_queue.cc:260] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.090862 2714 tablet_bootstrap.cc:492] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4: No bootstrap required, opened a new log
I20260929 00:56:53.090878 2713 consensus_queue.cc:260] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.090934 2714 ts_tablet_manager.cc:1397] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 00:56:53.091014 2716 ts_tablet_manager.cc:1428] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:53.091150 2713 ts_tablet_manager.cc:1428] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:56:53.091336 2714 raft_consensus.cc:348] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.091440 2714 raft_consensus.cc:374] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:53.091486 2714 raft_consensus.cc:729] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b706f745217546d3be5b4da860a15bc4, State: Initialized, Role: FOLLOWER
I20260929 00:56:53.091606 2714 consensus_queue.cc:260] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.091868 2714 ts_tablet_manager.cc:1428] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:56:53.200191 2718 raft_consensus.cc:482] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:53.200356 2718 raft_consensus.cc:504] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.200811 2718 leader_election.cc:288] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945), c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615)
I20260929 00:56:53.201344 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9521f40de341c096352935f63ff1ef" candidate_uuid: "b706f745217546d3be5b4da860a15bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daffb0824884f4190a8d26143ef33b6" is_pre_election: true
I20260929 00:56:53.201376 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9521f40de341c096352935f63ff1ef" candidate_uuid: "b706f745217546d3be5b4da860a15bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c206d894043048c9b85dfec92de3eb6a" is_pre_election: true
I20260929 00:56:53.201509 2415 raft_consensus.cc:2454] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b706f745217546d3be5b4da860a15bc4 in term 0.
I20260929 00:56:53.201529 2555 raft_consensus.cc:2454] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b706f745217546d3be5b4da860a15bc4 in term 0.
I20260929 00:56:53.201795 2208 leader_election.cc:302] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daffb0824884f4190a8d26143ef33b6, b706f745217546d3be5b4da860a15bc4; no voters:
I20260929 00:56:53.202066 2718 raft_consensus.cc:2781] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:53.202250 2718 raft_consensus.cc:482] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:53.202350 2718 raft_consensus.cc:3037] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:53.203557 2718 raft_consensus.cc:504] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.203948 2718 leader_election.cc:288] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [CANDIDATE]: Term 1 election: Requested vote from peers 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945), c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615)
I20260929 00:56:53.204269 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9521f40de341c096352935f63ff1ef" candidate_uuid: "b706f745217546d3be5b4da860a15bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daffb0824884f4190a8d26143ef33b6"
I20260929 00:56:53.204386 2415 raft_consensus.cc:3037] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:53.204371 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9521f40de341c096352935f63ff1ef" candidate_uuid: "b706f745217546d3be5b4da860a15bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c206d894043048c9b85dfec92de3eb6a"
I20260929 00:56:53.204476 2555 raft_consensus.cc:3037] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:53.205295 2555 raft_consensus.cc:2454] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b706f745217546d3be5b4da860a15bc4 in term 1.
I20260929 00:56:53.205561 2206 leader_election.cc:302] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b706f745217546d3be5b4da860a15bc4, c206d894043048c9b85dfec92de3eb6a; no voters:
I20260929 00:56:53.205591 2415 raft_consensus.cc:2454] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b706f745217546d3be5b4da860a15bc4 in term 1.
I20260929 00:56:53.205765 2718 raft_consensus.cc:2781] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:53.205997 2718 raft_consensus.cc:686] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 1 LEADER]: Becoming Leader. State: Replica: b706f745217546d3be5b4da860a15bc4, State: Running, Role: LEADER
I20260929 00:56:53.206306 2718 consensus_queue.cc:237] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } }
I20260929 00:56:53.208707 2117 catalog_manager.cc:5495] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b706f745217546d3be5b4da860a15bc4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b706f745217546d3be5b4da860a15bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706f745217546d3be5b4da860a15bc4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 } health_report { overall_health: UNKNOWN } } }
I20260929 00:56:53.244937 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:53.241675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.0.98.1:41635)
0929 00:56:53.241934 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:53.241941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:53.241953 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:56:53.242019 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:53.242021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:53.242070 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:53.242171 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:53.242430 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.242951 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.243700 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.243919 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.243989 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:53.244196 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:53.244628 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:53.244712 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:53.244770 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 00:56:53.244812 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":117,"threads_started":1}
I20260929 00:56:53.254499 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:53.251353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57453 (local address 127.0.98.3:35615)
0929 00:56:53.251452 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:53.251456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:53.251575 (+ 119us) server_negotiation.cc:407] Connection header received
0929 00:56:53.251719 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:53.251722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:53.251770 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:53.251847 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:53.252098 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.252578 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.253445 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.253678 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.253792 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:53.254062 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:53.254132 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:53.254232 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:53.254317 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 00:56:53.254365 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:56:53.254527 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:53.251166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35615 (local address 127.0.98.1:57453)
0929 00:56:53.251485 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:53.251504 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:53.251613 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:53.251866 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:53.251869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:53.251878 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:56:53.251992 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:53.251997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.252703 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.252707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:53.253329 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:53.253336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.253800 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.253804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:53.253894 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:53.253930 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:56:53.254234 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:56:53.254243 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:56:53.254308 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 00:56:53.254382 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
I20260929 00:56:53.259802 2415 raft_consensus.cc:1261] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Refusing update from remote peer b706f745217546d3be5b4da860a15bc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:53.259912 2555 raft_consensus.cc:1261] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Refusing update from remote peer b706f745217546d3be5b4da860a15bc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:53.260270 2733 consensus_queue.cc:1035] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daffb0824884f4190a8d26143ef33b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:53.260458 2718 consensus_queue.cc:1035] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c206d894043048c9b85dfec92de3eb6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:53.277776 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:53.274864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36653 (local address 127.0.98.1:41635)
0929 00:56:53.274986 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:53.274990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:53.275002 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:56:53.275044 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:53.275046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:53.275086 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:53.275170 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:53.275370 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.275854 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.276796 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.277115 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.277217 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:53.277476 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:53.277535 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:53.277588 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:53.277647 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 00:56:53.277686 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:56:53.277791 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:53.274495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41635 (local address 127.0.98.3:36653)
0929 00:56:53.274610 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:53.274624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:53.274714 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:53.275184 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:53.275187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:53.275194 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:56:53.275281 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:53.275287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.275985 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.275989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:53.276664 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:53.276673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:53.277222 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:53.277227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:53.277318 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:53.277377 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:56:53.277591 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:56:53.277596 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:56:53.277639 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 00:56:53.277702 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 00:56:53.278404 2685 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 00:56:54.833302 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:54.833482 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:54.835711 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 492c9800fd5a4bf4a552aade2d18cfdf: Sending DeleteTablet for 3 replicas of tablet 3e9521f40de341c096352935f63ff1ef
I20260929 00:56:54.836193 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9521f40de341c096352935f63ff1ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260929 00:56:54.836193 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9521f40de341c096352935f63ff1ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35690
I20260929 00:56:54.836275 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9521f40de341c096352935f63ff1ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41394
I20260929 00:56:54.836515 2769 tablet_replica.cc:330] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4: stopping tablet replica
I20260929 00:56:54.836645 2769 raft_consensus.cc:2229] T 3e9521f40de341c096352935f63ff1ef P b706f745217546d3be5b4da860a15bc4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:56:54.836651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 2190
I20260929 00:56:54.836694 2768 tablet_replica.cc:330] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: stopping tablet replica
I20260929 00:56:54.838714 2768 raft_consensus.cc:2229] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:54.838863 2768 raft_consensus.cc:2258] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:54.839135 2770 tablet_replica.cc:330] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: stopping tablet replica
I20260929 00:56:54.839321 2770 raft_consensus.cc:2229] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:54.839445 2770 raft_consensus.cc:2258] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:54.839583 2768 ts_tablet_manager.cc:1905] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:54.840065 2770 ts_tablet_manager.cc:1905] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:54.842314 2768 ts_tablet_manager.cc:1918] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:56:54.842396 2768 log.cc:1196] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/3e9521f40de341c096352935f63ff1ef
I20260929 00:56:54.842664 2770 ts_tablet_manager.cc:1918] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:56:54.842684 2768 ts_tablet_manager.cc:1939] T 3e9521f40de341c096352935f63ff1ef P 0daffb0824884f4190a8d26143ef33b6: Deleting consensus metadata
I20260929 00:56:54.842738 2770 log.cc:1196] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/3e9521f40de341c096352935f63ff1ef
I20260929 00:56:54.843010 2770 ts_tablet_manager.cc:1939] T 3e9521f40de341c096352935f63ff1ef P c206d894043048c9b85dfec92de3eb6a: Deleting consensus metadata
I20260929 00:56:54.843412 2104 catalog_manager.cc:4841] TS 0daffb0824884f4190a8d26143ef33b6 (127.0.98.2:32945): tablet 3e9521f40de341c096352935f63ff1ef (table test-table [id=dc61d975142048a892f1118aa5aed792]) successfully deleted
I20260929 00:56:54.843609 2102 catalog_manager.cc:4841] TS c206d894043048c9b85dfec92de3eb6a (127.0.98.3:35615): tablet 3e9521f40de341c096352935f63ff1ef (table test-table [id=dc61d975142048a892f1118aa5aed792]) successfully deleted
W20260929 00:56:54.843933 2103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41635 (error 108)
I20260929 00:56:54.844033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 2330
I20260929 00:56:54.850847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 2470
I20260929 00:56:54.852301 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:54.850954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41635 (local address 127.0.0.1:48302)
0929 00:56:54.852083 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:54.852172 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":103,"threads_started":1}
W20260929 00:56:54.852414 2103 catalog_manager.cc:4584] TS b706f745217546d3be5b4da860a15bc4 (127.0.98.1:41635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e9521f40de341c096352935f63ff1ef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41635: connect: Connection refused (error 111)
I20260929 00:56:54.857331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 2086
2026-09-29T00:56:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6378 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:56:54 dist-test-slave-30s6 krb5kdc[2775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:56:54 dist-test-slave-30s6 krb5kdc[2775](info): set up 2 sockets
Sep 29 00:56:54 dist-test-slave-30s6 krb5kdc[2775](info): commencing operation
krb5kdc: starting...
W20260929 00:56:55.898087 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:56:55 dist-test-slave-30s6 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:56:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:56:56.015008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46415
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:56.121488 2788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:56.121742 2788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:56.121788 2788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:56.125662 2788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:56.125711 2788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:56.125730 2788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:56.125746 2788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:56.125768 2788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:56.130769 2788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:56.131661 2788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:56.132592 2788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:56.140566 2796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:56.140575 2793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:56.140607 2788 server_base.cc:1034] running on GCE node
W20260929 00:56:56.140575 2794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:56.141350 2788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:56.142470 2788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:56.143610 2788 hybrid_clock.cc:648] HybridClock initialized: now 1790643416143584 us; error 39 us; skew 500 ppm
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:56.146353 2788 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:56:56.147204 2788 webserver.cc:460] Webserver started at http://127.0.98.62:33973/ using document root <none> and password file <none>
I20260929 00:56:56.147746 2788 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:56.147841 2788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:56.148053 2788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:56.150036 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "dd0650c032254241b128e565e5093d74"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "c9746c54be06ed00de9f70cad8dd643b"
server_key_iv: "8125cbb50d23b481856608c412a25ee4"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.150643 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd0650c032254241b128e565e5093d74"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "c9746c54be06ed00de9f70cad8dd643b"
server_key_iv: "8125cbb50d23b481856608c412a25ee4"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.154071 2788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:56.156350 2803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.157541 2788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:56.157675 2788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "dd0650c032254241b128e565e5093d74"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "c9746c54be06ed00de9f70cad8dd643b"
server_key_iv: "8125cbb50d23b481856608c412a25ee4"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.157785 2788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:56.179442 2788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:56.182418 2788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:56.182621 2788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:56.189587 2788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46415
I20260929 00:56:56.189604 2855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46415 every 8 connection(s)
I20260929 00:56:56.190608 2788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:56.193200 2856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:56.198223 2856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74: Bootstrap starting.
I20260929 00:56:56.200358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 2788
I20260929 00:56:56.200472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:56:56.200594 2856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:56.200755 392 external_mini_cluster.cc:1428] Setting key e35e467e942cc72af4b55ae0f2f74e11
I20260929 00:56:56.201458 2856 log.cc:824] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:56.203523 2856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74: No bootstrap required, opened a new log
I20260929 00:56:56.207861 2856 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0650c032254241b128e565e5093d74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46415 } }
I20260929 00:56:56.208194 2856 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:56.208274 2856 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0650c032254241b128e565e5093d74, State: Initialized, Role: FOLLOWER
I20260929 00:56:56.208840 2856 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0650c032254241b128e565e5093d74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46415 } }
I20260929 00:56:56.209019 2856 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:56:56.209102 2856 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:56:56.209227 2856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:56.210284 2856 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0650c032254241b128e565e5093d74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46415 } }
I20260929 00:56:56.210661 2856 leader_election.cc:302] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0650c032254241b128e565e5093d74; no voters:
I20260929 00:56:56.211028 2856 leader_election.cc:288] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:56:56.211134 2861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:56.211337 2861 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [term 1 LEADER]: Becoming Leader. State: Replica: dd0650c032254241b128e565e5093d74, State: Running, Role: LEADER
I20260929 00:56:56.211647 2861 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0650c032254241b128e565e5093d74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46415 } }
I20260929 00:56:56.212287 2856 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:56.212874 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0650c032254241b128e565e5093d74. Latest consensus state: current_term: 1 leader_uuid: "dd0650c032254241b128e565e5093d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0650c032254241b128e565e5093d74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46415 } } }
I20260929 00:56:56.212874 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd0650c032254241b128e565e5093d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0650c032254241b128e565e5093d74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46415 } } }
I20260929 00:56:56.213095 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:56.213357 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:56.213652 2867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:56:56.216410 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:56.201824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.0.98.62:46415)
0929 00:56:56.202214 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:56.202231 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:56.202278 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:56:56.202882 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:56.202897 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:56.203194 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:56.203462 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:56.204440 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.205402 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.206733 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.207175 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.207302 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:56.210258 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:56.210264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:56.210281 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:56.210315 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:56.212386 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:56.212934 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:56.212941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:56.212947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:56.213116 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:56.213523 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:56.213527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:56.213529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:56.214090 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:56.214994 (+ 904us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:56.215130 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 00:56:56.215361 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
I20260929 00:56:56.217264 2867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:56.222977 2867 catalog_manager.cc:1347] Generated new cluster ID: c0442aa642344e9887720420686aa391
I20260929 00:56:56.223068 2867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:56.230063 2867 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:56.231194 2867 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:56.239710 2867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd0650c032254241b128e565e5093d74: Generated new TSK 0
I20260929 00:56:56.240365 2867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:56:56.271296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46415
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:56.387691 2882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:56.387992 2882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:56.388085 2882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:56.392313 2882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:56.392400 2882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:56.392514 2882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:56:56.397749 2882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:56.398837 2882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:56.399835 2882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:56.409093 2887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:56.409114 2890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:56.409137 2888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:56.409315 2882 server_base.cc:1034] running on GCE node
I20260929 00:56:56.409783 2882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:56.410488 2882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:56.411671 2882 hybrid_clock.cc:648] HybridClock initialized: now 1790643416411659 us; error 25 us; skew 500 ppm
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:56.414502 2882 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:56:56.415294 2882 webserver.cc:460] Webserver started at http://127.0.98.1:46037/ using document root <none> and password file <none>
I20260929 00:56:56.415827 2882 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:56.415884 2882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:56.416095 2882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:56.418021 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "e552c9a690d54ac7b0aa609185cd2333"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "5fa26c851c234f805485fc59dbbff42d"
server_key_iv: "a9b8a85f695aa7007416cf3217f5b2a9"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.418622 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e552c9a690d54ac7b0aa609185cd2333"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "5fa26c851c234f805485fc59dbbff42d"
server_key_iv: "a9b8a85f695aa7007416cf3217f5b2a9"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.422433 2882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:56:56.424860 2897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.426110 2882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:56.426268 2882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "e552c9a690d54ac7b0aa609185cd2333"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "5fa26c851c234f805485fc59dbbff42d"
server_key_iv: "a9b8a85f695aa7007416cf3217f5b2a9"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.426385 2882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:56.439829 2882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:56.443856 2882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:56.444005 2882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:56.444514 2882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:56.445267 2882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:56.445313 2882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.445353 2882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:56.445374 2882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.453807 2882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34097
I20260929 00:56:56.453830 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34097 every 8 connection(s)
I20260929 00:56:56.454727 2882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:56:56.457813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 2882
I20260929 00:56:56.457953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:56:56.458276 392 external_mini_cluster.cc:1428] Setting key 758846af360965aa7eafd673f195de07
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:56:56.469502 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:56.456438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54683 (local address 127.0.98.62:46415)
0929 00:56:56.456564 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:56.456568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:56.457332 (+ 764us) server_negotiation.cc:407] Connection header received
0929 00:56:56.458947 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:56.458951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:56.459005 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:56.459108 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:56.460509 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.461114 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.462899 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.463129 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.463219 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:56.465963 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:56.465966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:56.465968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:56.465989 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:56.467430 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:56.467973 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:56.467978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:56.467980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:56.468060 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:56.468419 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:56.468424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:56.468426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:56.468602 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:56.468713 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:56.469247 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 00:56:56.469349 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 00:56:56.470431 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:56.456763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46415 (local address 127.0.98.1:54683)
0929 00:56:56.457180 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:56.457220 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:56.458719 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:56.459282 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:56.459290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:56.459854 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:56.460345 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:56.460358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.461274 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.461281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:56.462783 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:56.462797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.463289 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.463296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:56.463526 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:56.464154 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:56.464175 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:56.465799 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:56.467547 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:56.467553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:56.467567 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:56.467832 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:56.468157 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:56.468160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:56.468162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:56.468321 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:56.468714 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:56.468724 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:56.468967 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 00:56:56.469191 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 00:56:56.469506 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260929 00:56:56.471696 3011 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46415
I20260929 00:56:56.471961 3011 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:56.472546 3011 heartbeater.cc:511] Master 127.0.98.62:46415 requested a full tablet report, sending...
I20260929 00:56:56.474118 2820 ts_manager.cc:194] Registered new tserver with Master: e552c9a690d54ac7b0aa609185cd2333 (127.0.98.1:34097)
I20260929 00:56:56.475222 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:56:56.485653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46415
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:56.593346 3016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:56.593590 3016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:56.593629 3016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:56.597538 3016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:56.597586 3016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:56.597640 3016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:56:56.602607 3016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:56.603505 3016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:56.604413 3016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:56.613896 3021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:56.613891 3022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:56.614010 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:56.614323 3016 server_base.cc:1034] running on GCE node
I20260929 00:56:56.614651 3016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:56.615244 3016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:56.616413 3016 hybrid_clock.cc:648] HybridClock initialized: now 1790643416616387 us; error 38 us; skew 500 ppm
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:56.619066 3016 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:56:56.619911 3016 webserver.cc:460] Webserver started at http://127.0.98.2:42473/ using document root <none> and password file <none>
I20260929 00:56:56.620445 3016 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:56.620527 3016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:56.620761 3016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:56.622721 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "b1a330fb654e405f8b9e2b0ca13f5fdd"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "4a3e7caee086d07f80239fbad4f19db8"
server_key_iv: "bdd66742894e3b9384b184bb65c77460"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.623257 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b1a330fb654e405f8b9e2b0ca13f5fdd"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "4a3e7caee086d07f80239fbad4f19db8"
server_key_iv: "bdd66742894e3b9384b184bb65c77460"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.626662 3016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:56:56.628834 3031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.630004 3016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:56.630138 3016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "b1a330fb654e405f8b9e2b0ca13f5fdd"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "4a3e7caee086d07f80239fbad4f19db8"
server_key_iv: "bdd66742894e3b9384b184bb65c77460"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.630311 3016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:56.649464 3016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:56.652676 3016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:56.652873 3016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:56.653436 3016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:56.654277 3016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:56.654340 3016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.654410 3016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:56.654448 3016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.663117 3016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40839
I20260929 00:56:56.663141 3144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40839 every 8 connection(s)
I20260929 00:56:56.663969 3016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:56:56.671561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 3016
I20260929 00:56:56.671690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:56:56.671965 392 external_mini_cluster.cc:1428] Setting key 60145684caacfa55aa09b590fedbb792
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:56.677245 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:56.665595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47515 (local address 127.0.98.62:46415)
0929 00:56:56.665732 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:56.665737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:56.666643 (+ 906us) server_negotiation.cc:407] Connection header received
0929 00:56:56.667945 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:56.667948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:56.667992 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:56.668066 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:56.669140 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.669636 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.670768 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.670971 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.671046 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:56.673655 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:56.673658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:56.673660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:56.673684 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:56.675254 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:56.675792 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:56.675796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:56.675797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:56.675868 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:56.676197 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:56.676202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:56.676204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:56.676382 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:56.676486 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:56.676966 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 00:56:56.677077 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:56:56.678046 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:56.666038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46415 (local address 127.0.98.2:47515)
0929 00:56:56.666505 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:56.666545 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:56.667761 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:56.668203 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:56.668209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:56.668554 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:56.669021 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:56.669033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.669734 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.669736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:56.670671 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:56.670680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.671089 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.671092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:56.671238 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:56.671852 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:56.671867 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:56.673486 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:56.675390 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:56.675395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:56.675407 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:56.675648 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:56.675961 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:56.675966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:56.675968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:56.676102 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:56.676484 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:56.676491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:56.676737 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 00:56:56.676924 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 00:56:56.677238 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":166,"threads_started":1}
I20260929 00:56:56.679221 3145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46415
I20260929 00:56:56.679487 3145 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:56.680105 3145 heartbeater.cc:511] Master 127.0.98.62:46415 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:56:56.681242 2820 ts_manager.cc:194] Registered new tserver with Master: b1a330fb654e405f8b9e2b0ca13f5fdd (127.0.98.2:40839)
I20260929 00:56:56.681782 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:56:56.697132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46415
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:56.809674 3150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:56.809916 3150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:56.809958 3150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:56.814339 3150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:56.814397 3150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:56.814502 3150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:56:56.819700 3150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:56.820750 3150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:56.821731 3150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:56.830933 3155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:56.830921 3158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:56.830925 3156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:56.831326 3150 server_base.cc:1034] running on GCE node
I20260929 00:56:56.831698 3150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:56.832295 3150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:56.833492 3150 hybrid_clock.cc:648] HybridClock initialized: now 1790643416833480 us; error 25 us; skew 500 ppm
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:56.836400 3150 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:56:56.837283 3150 webserver.cc:460] Webserver started at http://127.0.98.3:35453/ using document root <none> and password file <none>
I20260929 00:56:56.837831 3150 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:56.837888 3150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:56.838114 3150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:56.840101 3150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "b04bb1e072104451bd5d70302e8e9e8f"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "58c21eacd042e10f80788cad8cf58142"
server_key_iv: "787ff83a9d7bed832f80cb8ba747fc97"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.840663 3150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b04bb1e072104451bd5d70302e8e9e8f"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "58c21eacd042e10f80788cad8cf58142"
server_key_iv: "787ff83a9d7bed832f80cb8ba747fc97"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.844235 3150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 00:56:56.846557 3165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.847723 3150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:56:56.847836 3150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "b04bb1e072104451bd5d70302e8e9e8f"
format_stamp: "Formatted at 2026-09-29 00:56:56 on dist-test-slave-30s6"
server_key: "58c21eacd042e10f80788cad8cf58142"
server_key_iv: "787ff83a9d7bed832f80cb8ba747fc97"
server_key_version: "encryptionkey@0"
I20260929 00:56:56.847936 3150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:56.866777 3150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:56.869946 3150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:56.870137 3150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:56.870718 3150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:56.871472 3150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:56.871516 3150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.871583 3150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:56.871619 3150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:56.880311 3150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36585
I20260929 00:56:56.880365 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36585 every 8 connection(s)
I20260929 00:56:56.881179 3150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:56:56.883194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 3150
I20260929 00:56:56.883319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:56:56.883641 392 external_mini_cluster.cc:1428] Setting key 72e83486fa68cb25aa52a687a6dfab68
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:56.894099 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:56.882876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34575 (local address 127.0.98.62:46415)
0929 00:56:56.883015 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:56.883021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:56.883864 (+ 843us) server_negotiation.cc:407] Connection header received
0929 00:56:56.885545 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:56.885548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:56.885600 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:56.885691 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:56.887024 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.887524 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.888585 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.888787 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.888861 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:56.891092 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:56.891095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:56.891097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:56.891119 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:56.892517 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:56.892923 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:56.892925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:56.892926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:56.892994 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:56.893274 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:56.893276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:56.893277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:56.893430 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:56.893501 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:56.893902 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 00:56:56.893985 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:56:56.894899 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:56.883184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46415 (local address 127.0.98.3:34575)
0929 00:56:56.883696 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:56.883752 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:56.885313 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:56.885862 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:56.885869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:56.886265 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:56.886908 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:56.886922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.887619 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.887621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:56.888489 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:56.888497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:56.888903 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:56.888906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:56.889034 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:56.889549 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:56.889564 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:56.890958 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:56.892605 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:56.892610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:56.892620 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:56.892844 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:56.893067 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:56.893069 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:56.893070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:56.893193 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:56.893510 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:56.893514 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:56.893717 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 00:56:56.893861 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 00:56:56.894085 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":151,"threads_started":1}
I20260929 00:56:56.896128 3279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46415
I20260929 00:56:56.896354 3279 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:56.896888 3279 heartbeater.cc:511] Master 127.0.98.62:46415 requested a full tablet report, sending...
I20260929 00:56:56.897835 2820 ts_manager.cc:194] Registered new tserver with Master: b04bb1e072104451bd5d70302e8e9e8f (127.0.98.3:36585)
I20260929 00:56:56.898458 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34575
I20260929 00:56:56.908901 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:56:56.920768 2860 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:56.919579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.0.98.62:46415)
0929 00:56:56.919702 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:56.919705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:56.919846 (+ 141us) server_negotiation.cc:407] Connection header received
0929 00:56:56.919948 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:56.919950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:56.919990 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:56.920061 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:56.920668 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42760: BlockingRecv error: recv got EOF from 127.0.0.1:42760 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:56:56.921731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 2882
I20260929 00:56:56.927052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 3016
I20260929 00:56:56.932273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 3150
I20260929 00:56:56.937568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 2788
2026-09-29T00:56:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2079 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[3291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[3291](info): set up 2 sockets
Sep 29 00:56:56 dist-test-slave-30s6 krb5kdc[3291](info): commencing operation
krb5kdc: starting...
W20260929 00:56:57.974450 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:56:58 dist-test-slave-30s6 krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:56:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:56:58.102025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43039
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:58.215924 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:58.216212 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:58.216253 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:58.220273 3304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:58.220329 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:58.220348 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:58.220366 3304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:58.220381 3304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:58.225700 3304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:58.226773 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:58.227752 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:58.236198 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:58.236161 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:58.236161 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:58.236433 3304 server_base.cc:1034] running on GCE node
I20260929 00:56:58.237111 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:58.238344 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:58.239529 3304 hybrid_clock.cc:648] HybridClock initialized: now 1790643418239507 us; error 38 us; skew 500 ppm
Sep 29 00:56:58 dist-test-slave-30s6 krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:58.242334 3304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:56:58.243186 3304 webserver.cc:460] Webserver started at http://127.0.98.62:34877/ using document root <none> and password file <none>
I20260929 00:56:58.243706 3304 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:58.243767 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:58.243928 3304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:58.245991 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "7cb9fa8e6fb44ce990aafe5ec9806940"
format_stamp: "Formatted at 2026-09-29 00:56:58 on dist-test-slave-30s6"
server_key: "fa45b59e145654e5e6f3698b95c24cf8"
server_key_iv: "c4cb55b78a9124c6b49b5a21337fd82f"
server_key_version: "encryptionkey@0"
I20260929 00:56:58.246595 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "7cb9fa8e6fb44ce990aafe5ec9806940"
format_stamp: "Formatted at 2026-09-29 00:56:58 on dist-test-slave-30s6"
server_key: "fa45b59e145654e5e6f3698b95c24cf8"
server_key_iv: "c4cb55b78a9124c6b49b5a21337fd82f"
server_key_version: "encryptionkey@0"
I20260929 00:56:58.250267 3304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:58.252612 3319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:58.253876 3304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:56:58.253976 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "7cb9fa8e6fb44ce990aafe5ec9806940"
format_stamp: "Formatted at 2026-09-29 00:56:58 on dist-test-slave-30s6"
server_key: "fa45b59e145654e5e6f3698b95c24cf8"
server_key_iv: "c4cb55b78a9124c6b49b5a21337fd82f"
server_key_version: "encryptionkey@0"
I20260929 00:56:58.254065 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:58.270283 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:58.273308 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:58.273460 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:58.280483 3304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43039
I20260929 00:56:58.280773 3371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43039 every 8 connection(s)
I20260929 00:56:58.281445 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:58.282800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 3304
I20260929 00:56:58.282927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:56:58.283260 392 external_mini_cluster.cc:1428] Setting key d06f9fb43e7c7ecfccd943a1bfe866d2
I20260929 00:56:58.284803 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:58.291436 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940: Bootstrap starting.
Sep 29 00:56:58 dist-test-slave-30s6 krb5kdc[3291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:58.294153 3372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:58.295086 3372 log.cc:824] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:58.297381 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940: No bootstrap required, opened a new log
I20260929 00:56:58.298285 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:58.284556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.0.98.62:43039)
0929 00:56:58.284955 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:58.284970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:58.285003 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:56:58.285631 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:58.285647 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:58.285957 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:58.286294 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:58.287173 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:58.288128 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:58.289314 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:58.289911 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:58.290025 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:58.292956 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:58.292963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:58.292979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:58.293011 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:58.295096 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:58.295589 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:58.295594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:58.295598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:58.295713 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:58.296015 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:58.296019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:58.296021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:58.296581 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:58.296779 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:58.297012 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 00:56:58.297307 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":134,"threads_started":1}
I20260929 00:56:58.301180 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:56:58.301800 3372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb9fa8e6fb44ce990aafe5ec9806940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43039 } }
I20260929 00:56:58.302034 3372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:58.302116 3372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb9fa8e6fb44ce990aafe5ec9806940, State: Initialized, Role: FOLLOWER
I20260929 00:56:58.302610 3372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb9fa8e6fb44ce990aafe5ec9806940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43039 } }
I20260929 00:56:58.302766 3372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:56:58.302866 3372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:56:58.303013 3372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:58.303999 3372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb9fa8e6fb44ce990aafe5ec9806940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43039 } }
I20260929 00:56:58.304356 3372 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cb9fa8e6fb44ce990aafe5ec9806940; no voters:
I20260929 00:56:58.304708 3372 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:56:58.304796 3378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:58.304987 3378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [term 1 LEADER]: Becoming Leader. State: Replica: 7cb9fa8e6fb44ce990aafe5ec9806940, State: Running, Role: LEADER
I20260929 00:56:58.305356 3378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb9fa8e6fb44ce990aafe5ec9806940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43039 } }
I20260929 00:56:58.306064 3372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:58.307958 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb9fa8e6fb44ce990aafe5ec9806940. Latest consensus state: current_term: 1 leader_uuid: "7cb9fa8e6fb44ce990aafe5ec9806940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb9fa8e6fb44ce990aafe5ec9806940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43039 } } }
I20260929 00:56:58.308084 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:58.308475 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cb9fa8e6fb44ce990aafe5ec9806940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb9fa8e6fb44ce990aafe5ec9806940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43039 } } }
I20260929 00:56:58.308706 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:56:58.308907 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:58.311522 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:58.318041 3392 catalog_manager.cc:1347] Generated new cluster ID: d180432edc254ddbbc70aeeb9dea1972
I20260929 00:56:58.318113 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:58.323206 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:58.318548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.0.98.62:43039)
0929 00:56:58.318673 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:58.318678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:58.318922 (+ 244us) server_negotiation.cc:407] Connection header received
0929 00:56:58.319074 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:58.319077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:58.319124 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:58.319214 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:58.319748 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:58.320273 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:58.321101 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:58.321358 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:58.321465 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:58.322344 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:58.322348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:58.322351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:56:58.322776 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:58.322804 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:56:58.322818 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:58.322916 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:58.323005 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 00:56:58.323063 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:56:58.333817 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:58.334836 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:58.341281 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cb9fa8e6fb44ce990aafe5ec9806940: Generated new TSK 0
I20260929 00:56:58.341985 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:56:58.351954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 3304
2026-09-29T00:56:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1412 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:56:58 dist-test-slave-30s6 krb5kdc[3405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:56:58 dist-test-slave-30s6 krb5kdc[3405](info): set up 2 sockets
Sep 29 00:56:58 dist-test-slave-30s6 krb5kdc[3405](info): commencing operation
krb5kdc: starting...
W20260929 00:56:59.402282 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:56:59 dist-test-slave-30s6 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:56:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:56:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:56:59.530709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:59.642040 3418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:59.642330 3418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:59.642373 3418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:59.646373 3418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:59.646425 3418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:59.646451 3418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:59.646468 3418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:59.646484 3418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:59.651626 3418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:59.652654 3418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:59.653679 3418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:59.662139 3423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:59.662268 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:59.662250 3418 server_base.cc:1034] running on GCE node
W20260929 00:56:59.662252 3424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:59.663002 3418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:59.664043 3418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:59.665217 3418 hybrid_clock.cc:648] HybridClock initialized: now 1790643419665201 us; error 35 us; skew 500 ppm
Sep 29 00:56:59 dist-test-slave-30s6 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:59.668007 3418 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:56:59.668881 3418 webserver.cc:460] Webserver started at http://127.0.98.62:40835/ using document root <none> and password file <none>
I20260929 00:56:59.669380 3418 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:59.669435 3418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:59.669595 3418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:59.671666 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "0f41e02302aa4bf4a843545e94fc361b"
format_stamp: "Formatted at 2026-09-29 00:56:59 on dist-test-slave-30s6"
server_key: "661372402d6f9f34c22a10d7750b4431"
server_key_iv: "5a893c8d62f87d80a2384ebb69587c9a"
server_key_version: "encryptionkey@0"
I20260929 00:56:59.672187 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f41e02302aa4bf4a843545e94fc361b"
format_stamp: "Formatted at 2026-09-29 00:56:59 on dist-test-slave-30s6"
server_key: "661372402d6f9f34c22a10d7750b4431"
server_key_iv: "5a893c8d62f87d80a2384ebb69587c9a"
server_key_version: "encryptionkey@0"
I20260929 00:56:59.675899 3418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 00:56:59.678381 3433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:59.679662 3418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:56:59.679767 3418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "0f41e02302aa4bf4a843545e94fc361b"
format_stamp: "Formatted at 2026-09-29 00:56:59 on dist-test-slave-30s6"
server_key: "661372402d6f9f34c22a10d7750b4431"
server_key_iv: "5a893c8d62f87d80a2384ebb69587c9a"
server_key_version: "encryptionkey@0"
I20260929 00:56:59.679929 3418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:59.691902 3418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:59.694962 3418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:59.695164 3418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:59.702023 3418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44837
I20260929 00:56:59.702364 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44837 every 8 connection(s)
I20260929 00:56:59.703051 3418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:59.705732 3486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:59.706920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 3418
I20260929 00:56:59.707010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:56:59.707296 392 external_mini_cluster.cc:1428] Setting key 4c39586a0745b51ee8003afd5f216e1b
I20260929 00:56:59.712364 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b: Bootstrap starting.
I20260929 00:56:59.714846 3486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:59.715540 3486 log.cc:824] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:56:59 dist-test-slave-30s6 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:56:59.717572 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b: No bootstrap required, opened a new log
I20260929 00:56:59.722374 3486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f41e02302aa4bf4a843545e94fc361b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44837 } }
I20260929 00:56:59.722636 3486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:59.722641 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:59.708739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.0.98.62:44837)
0929 00:56:59.709145 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:59.709163 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:59.709205 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:56:59.710009 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:59.710030 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:59.710475 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:59.710860 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:59.711541 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:59.712941 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:59.713978 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:59.714665 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:59.714831 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:59.717537 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:59.717543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:59.717560 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:59.717592 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:59.719581 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:59.720007 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:59.720016 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:59.720023 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:59.720193 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:59.720460 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:59.720463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:59.720465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:59.720951 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:59.721155 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:59.721347 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 00:56:59.721661 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260929 00:56:59.722713 3486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f41e02302aa4bf4a843545e94fc361b, State: Initialized, Role: FOLLOWER
I20260929 00:56:59.723142 3486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f41e02302aa4bf4a843545e94fc361b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44837 } }
I20260929 00:56:59.723296 3486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:56:59.723387 3486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:56:59.723484 3486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:59.724372 3486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f41e02302aa4bf4a843545e94fc361b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44837 } }
I20260929 00:56:59.724707 3486 leader_election.cc:302] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f41e02302aa4bf4a843545e94fc361b; no voters:
I20260929 00:56:59.725051 3486 leader_election.cc:288] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:56:59.725298 3491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:59.725591 3491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [term 1 LEADER]: Becoming Leader. State: Replica: 0f41e02302aa4bf4a843545e94fc361b, State: Running, Role: LEADER
I20260929 00:56:59.725976 3491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f41e02302aa4bf4a843545e94fc361b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44837 } }
I20260929 00:56:59.726291 3486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:59.727828 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f41e02302aa4bf4a843545e94fc361b. Latest consensus state: current_term: 1 leader_uuid: "0f41e02302aa4bf4a843545e94fc361b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f41e02302aa4bf4a843545e94fc361b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44837 } } }
I20260929 00:56:59.727978 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:59.728402 3505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:56:59.728353 3492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f41e02302aa4bf4a843545e94fc361b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f41e02302aa4bf4a843545e94fc361b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44837 } } }
I20260929 00:56:59.728932 3492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:59.731801 3505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:59.737169 3505 catalog_manager.cc:1347] Generated new cluster ID: 74ef9e95687c4362bbe13cbe87d372df
I20260929 00:56:59.737234 3505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:59.753604 3505 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:59.754577 3505 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:59.761209 3505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f41e02302aa4bf4a843545e94fc361b: Generated new TSK 0
I20260929 00:56:59.761915 3505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:56:59.797080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44837
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:59.909571 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:59.909876 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:59.909922 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:59.914039 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:59.914108 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:59.914223 3512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:56:59.919596 3512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:59.920717 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:59.921810 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:59.931949 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:59.931949 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:59.931949 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:59.932457 3512 server_base.cc:1034] running on GCE node
I20260929 00:56:59.932893 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:59.933580 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:59.934772 3512 hybrid_clock.cc:648] HybridClock initialized: now 1790643419934755 us; error 30 us; skew 500 ppm
Sep 29 00:56:59 dist-test-slave-30s6 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:59.944840 3512 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:56:59.945866 3512 webserver.cc:460] Webserver started at http://127.0.98.1:40463/ using document root <none> and password file <none>
I20260929 00:56:59.946485 3512 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:59.946551 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:59.946768 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:59.948865 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f42da672f3b4574bf5a14cdf02a7df1"
format_stamp: "Formatted at 2026-09-29 00:56:59 on dist-test-slave-30s6"
server_key: "b6d9bdf312fdf855048eb1d649992c6b"
server_key_iv: "b36e966bc003900effb6d4dc717cdaa5"
server_key_version: "encryptionkey@0"
I20260929 00:56:59.949432 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f42da672f3b4574bf5a14cdf02a7df1"
format_stamp: "Formatted at 2026-09-29 00:56:59 on dist-test-slave-30s6"
server_key: "b6d9bdf312fdf855048eb1d649992c6b"
server_key_iv: "b36e966bc003900effb6d4dc717cdaa5"
server_key_version: "encryptionkey@0"
I20260929 00:56:59.953193 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 00:56:59.955646 3527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:59.956791 3512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:56:59.956918 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "2f42da672f3b4574bf5a14cdf02a7df1"
format_stamp: "Formatted at 2026-09-29 00:56:59 on dist-test-slave-30s6"
server_key: "b6d9bdf312fdf855048eb1d649992c6b"
server_key_iv: "b36e966bc003900effb6d4dc717cdaa5"
server_key_version: "encryptionkey@0"
I20260929 00:56:59.957042 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:59.971130 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:59.974570 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:59.974772 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:59.975368 3512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:59.976191 3512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:59.976265 3512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:59.976333 3512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:59.976382 3512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:59.985590 3512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38705
I20260929 00:56:59.985616 3640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38705 every 8 connection(s)
I20260929 00:56:59.986580 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:56:59.993953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 3512
I20260929 00:56:59.994091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:56:59.994444 392 external_mini_cluster.cc:1428] Setting key 9cf397d938d7d27f2ea49bfc63b30641
Sep 29 00:56:59 dist-test-slave-30s6 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:00.002112 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:59.988375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37091 (local address 127.0.98.62:44837)
0929 00:56:59.988518 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:59.988523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:59.989428 (+ 905us) server_negotiation.cc:407] Connection header received
0929 00:56:59.990840 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:59.990842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:59.990887 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:59.990963 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:59.992101 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:59.992665 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:59.994621 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:59.994890 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:59.995020 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:59.998118 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:59.998121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:59.998124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:59.998150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:59.999667 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:00.000361 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:00.000364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:00.000365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:00.000437 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:00.000863 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:00.000865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:00.000867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:00.001047 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:00.001119 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:00.001870 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 00:57:00.001975 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:57:00.003477 3643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:59.988837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44837 (local address 127.0.98.1:37091)
0929 00:56:59.989278 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:59.989322 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:59.990647 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:59.991128 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:59.991135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:59.991511 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:59.991976 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:59.991989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:59.992811 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:59.992815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:59.994458 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:59.994473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:59.995087 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:59.995091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:59.995299 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:59.995951 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:59.995973 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:59.997902 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:59.999796 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:59.999805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:59.999827 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:00.000257 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:00.000540 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:00.000545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:00.000549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:00.000761 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:00.001157 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:00.001167 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:00.001579 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 00:57:00.001845 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 00:57:00.002280 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:57:00.004886 3641 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44837
I20260929 00:57:00.005162 3641 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:00.005674 3641 heartbeater.cc:511] Master 127.0.98.62:44837 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 00:57:00.007467 3450 ts_manager.cc:194] Registered new tserver with Master: 2f42da672f3b4574bf5a14cdf02a7df1 (127.0.98.1:38705)
I20260929 00:57:00.009064 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:57:00.023296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44837
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:00.133169 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:00.133445 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:00.133492 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:00.137554 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:00.137629 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:00.137729 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:00.143174 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:00.144299 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:00.145362 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:00.155179 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:00.155262 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:00.155179 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:00.155429 3646 server_base.cc:1034] running on GCE node
I20260929 00:57:00.155966 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:00.156629 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:00.157819 3646 hybrid_clock.cc:648] HybridClock initialized: now 1790643420157791 us; error 53 us; skew 500 ppm
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:00.160776 3646 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:57:00.161640 3646 webserver.cc:460] Webserver started at http://127.0.98.2:44107/ using document root <none> and password file <none>
I20260929 00:57:00.162163 3646 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:00.162289 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:00.162462 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:00.164490 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "ead10f0c11ab4bc69bac4c1e895951a9"
format_stamp: "Formatted at 2026-09-29 00:57:00 on dist-test-slave-30s6"
server_key: "3db692eb3055872980dabe4a963ca06f"
server_key_iv: "eb1ffbfe0b9c2264a10ac51f17b854ef"
server_key_version: "encryptionkey@0"
I20260929 00:57:00.165132 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ead10f0c11ab4bc69bac4c1e895951a9"
format_stamp: "Formatted at 2026-09-29 00:57:00 on dist-test-slave-30s6"
server_key: "3db692eb3055872980dabe4a963ca06f"
server_key_iv: "eb1ffbfe0b9c2264a10ac51f17b854ef"
server_key_version: "encryptionkey@0"
I20260929 00:57:00.168974 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:57:00.171401 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:00.172626 3646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:57:00.172747 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "ead10f0c11ab4bc69bac4c1e895951a9"
format_stamp: "Formatted at 2026-09-29 00:57:00 on dist-test-slave-30s6"
server_key: "3db692eb3055872980dabe4a963ca06f"
server_key_iv: "eb1ffbfe0b9c2264a10ac51f17b854ef"
server_key_version: "encryptionkey@0"
I20260929 00:57:00.172930 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:00.209218 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:00.212743 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:00.212957 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:00.213581 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:00.214484 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:00.214531 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:00.214600 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:00.214643 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:00.223742 3646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41021
I20260929 00:57:00.223760 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41021 every 8 connection(s)
I20260929 00:57:00.224646 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:00.229732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 3646
I20260929 00:57:00.229847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:00.230126 392 external_mini_cluster.cc:1428] Setting key 179cb8c11a7fad03aaf09460bc168a45
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:57:00.240180 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:00.226517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57785 (local address 127.0.98.62:44837)
0929 00:57:00.226657 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:00.226662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:00.227630 (+ 968us) server_negotiation.cc:407] Connection header received
0929 00:57:00.229198 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:00.229201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:00.229244 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:00.229319 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:00.230862 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.231374 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.232494 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.232878 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.232999 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:00.235972 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:00.235979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:00.235986 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:00.236021 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:00.237854 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:00.238480 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:00.238485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:00.238489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:00.238587 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:00.238939 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:00.238944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:00.238946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:00.239193 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:00.239306 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:00.239821 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 00:57:00.239954 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:57:00.240934 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:00.226948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44837 (local address 127.0.98.2:57785)
0929 00:57:00.227487 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:00.227530 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:00.229001 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:00.229526 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:00.229535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:00.230066 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:00.230714 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:00.230729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.231510 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.231512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:00.232362 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:00.232371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.233094 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.233101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:00.233355 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:00.234036 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:00.234060 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:00.235775 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:00.237992 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:00.237997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:00.238012 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:00.238368 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:00.238691 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:00.238693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:00.238695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:00.238829 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:00.239307 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:00.239312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:00.239565 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 00:57:00.239714 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 00:57:00.239973 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":143,"threads_started":1}
I20260929 00:57:00.242226 3775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44837
I20260929 00:57:00.242497 3775 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:57:00.242997 3775 heartbeater.cc:511] Master 127.0.98.62:44837 requested a full tablet report, sending...
I20260929 00:57:00.244041 3450 ts_manager.cc:194] Registered new tserver with Master: ead10f0c11ab4bc69bac4c1e895951a9 (127.0.98.2:41021)
I20260929 00:57:00.244611 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:57:00.257720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44837
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:00.369264 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:00.369555 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:00.369602 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:00.373737 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:00.373808 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:00.373871 3780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:00.379403 3780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:00.380450 3780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:00.381445 3780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:00.391088 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:00.391112 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:00.391091 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:00.391292 3780 server_base.cc:1034] running on GCE node
I20260929 00:57:00.391954 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:00.392531 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:00.393690 3780 hybrid_clock.cc:648] HybridClock initialized: now 1790643420393677 us; error 28 us; skew 500 ppm
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:00.396445 3780 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:57:00.397236 3780 webserver.cc:460] Webserver started at http://127.0.98.3:46125/ using document root <none> and password file <none>
I20260929 00:57:00.397743 3780 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:00.397799 3780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:00.397956 3780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:00.399987 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "89f7d662d0d34be8846bc15776541b7e"
format_stamp: "Formatted at 2026-09-29 00:57:00 on dist-test-slave-30s6"
server_key: "4463b203fee5e466b62f63b4aeb0ed1d"
server_key_iv: "0949393dcd5c7c227e48941ed0656a14"
server_key_version: "encryptionkey@0"
I20260929 00:57:00.400488 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "89f7d662d0d34be8846bc15776541b7e"
format_stamp: "Formatted at 2026-09-29 00:57:00 on dist-test-slave-30s6"
server_key: "4463b203fee5e466b62f63b4aeb0ed1d"
server_key_iv: "0949393dcd5c7c227e48941ed0656a14"
server_key_version: "encryptionkey@0"
I20260929 00:57:00.404140 3780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:57:00.406451 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:00.407696 3780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:57:00.407819 3780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "89f7d662d0d34be8846bc15776541b7e"
format_stamp: "Formatted at 2026-09-29 00:57:00 on dist-test-slave-30s6"
server_key: "4463b203fee5e466b62f63b4aeb0ed1d"
server_key_iv: "0949393dcd5c7c227e48941ed0656a14"
server_key_version: "encryptionkey@0"
I20260929 00:57:00.407922 3780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:00.449469 3780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:00.452981 3780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:00.453192 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:00.453887 3780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:00.454809 3780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:00.454854 3780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:00.454916 3780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:00.454952 3780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:00.465168 3780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39809
I20260929 00:57:00.465188 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39809 every 8 connection(s)
I20260929 00:57:00.466109 3780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:00.474826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 3780
I20260929 00:57:00.474961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:00.475349 392 external_mini_cluster.cc:1428] Setting key 6e499829d4cfce4c9c05499e849ac737
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:00.480861 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:00.468063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49975 (local address 127.0.98.62:44837)
0929 00:57:00.468218 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:00.468221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:00.469118 (+ 897us) server_negotiation.cc:407] Connection header received
0929 00:57:00.470660 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:00.470664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:00.470727 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:00.470839 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:00.472063 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.472610 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.473817 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.474047 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.474169 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:00.477452 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:00.477456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:00.477458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:00.477484 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:00.478848 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:00.479370 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:00.479374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:00.479375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:00.479457 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:00.479804 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:00.479808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:00.479810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:00.479990 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:00.480097 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:00.480587 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 00:57:00.480705 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:57:00.481678 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:00.468471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44837 (local address 127.0.98.3:49975)
0929 00:57:00.468953 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:00.468999 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:00.470423 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:00.471017 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:00.471024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:00.471412 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:00.471905 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:00.471919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.472748 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.472763 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:00.473670 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:00.473680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.474241 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.474247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:00.474432 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:00.475170 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:00.475196 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:00.477250 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:00.478970 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:00.478976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:00.478992 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:00.479251 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:00.479556 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:00.479561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:00.479563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:00.479706 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:00.480095 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:00.480102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:00.480351 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 00:57:00.480537 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 00:57:00.480874 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":125,"threads_started":1}
I20260929 00:57:00.482769 3909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44837
I20260929 00:57:00.482986 3909 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:00.483408 3909 heartbeater.cc:511] Master 127.0.98.62:44837 requested a full tablet report, sending...
I20260929 00:57:00.484426 3450 ts_manager.cc:194] Registered new tserver with Master: 89f7d662d0d34be8846bc15776541b7e (127.0.98.3:39809)
I20260929 00:57:00.484995 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49975
I20260929 00:57:00.489185 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643420, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643420, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:00.514734 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:00.505553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.0.98.62:44837)
0929 00:57:00.505691 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:00.505695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:00.505876 (+ 181us) server_negotiation.cc:407] Connection header received
0929 00:57:00.506047 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:00.506050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:00.506094 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:00.506186 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:00.506885 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.507405 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.508282 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:00.508507 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:00.508612 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:00.511231 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:00.511234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:00.511236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:00.511262 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:00.512794 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:00.513265 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:00.513270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:00.513272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:00.513379 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:00.513687 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:00.513691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:00.513693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:00.513950 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:00.514105 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:00.514422 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 00:57:00.514543 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 00:57:00.515164 3450 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34984
I20260929 00:57:00.516011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 3512
I20260929 00:57:00.521773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 3646
I20260929 00:57:00.527209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 3780
I20260929 00:57:00.532328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 3418
2026-09-29T00:57:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2182 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3921](info): set up 2 sockets
Sep 29 00:57:00 dist-test-slave-30s6 krb5kdc[3921](info): commencing operation
krb5kdc: starting...
W20260929 00:57:01.570348 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:01 dist-test-slave-30s6 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:57:01.698342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36415
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:01.808279 3934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:01.808605 3934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:01.808689 3934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:01.812876 3934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:01.812984 3934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:01.813024 3934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:01.813077 3934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:01.813114 3934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:01.818560 3934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:01.819811 3934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:01.820899 3934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:01.829574 3940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:01.829607 3939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:01.829720 3934 server_base.cc:1034] running on GCE node
W20260929 00:57:01.829916 3942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:01.830598 3934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:01.831779 3934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:01.832959 3934 hybrid_clock.cc:648] HybridClock initialized: now 1790643421832928 us; error 49 us; skew 500 ppm
Sep 29 00:57:01 dist-test-slave-30s6 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:01.836056 3934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:57:01.837023 3934 webserver.cc:460] Webserver started at http://127.0.98.62:42245/ using document root <none> and password file <none>
I20260929 00:57:01.837622 3934 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:01.837685 3934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:01.837908 3934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:01.840044 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "33e9aa22a4204b389c97f7f4bbbc8b11"
format_stamp: "Formatted at 2026-09-29 00:57:01 on dist-test-slave-30s6"
server_key: "5be3ad2a4c82bded9762a6e0963125ab"
server_key_iv: "cebeba361485c41739fc2f1ddd8b1aa6"
server_key_version: "encryptionkey@0"
I20260929 00:57:01.840613 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "33e9aa22a4204b389c97f7f4bbbc8b11"
format_stamp: "Formatted at 2026-09-29 00:57:01 on dist-test-slave-30s6"
server_key: "5be3ad2a4c82bded9762a6e0963125ab"
server_key_iv: "cebeba361485c41739fc2f1ddd8b1aa6"
server_key_version: "encryptionkey@0"
I20260929 00:57:01.844554 3934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:01.847091 3949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:01.848380 3934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:01.848510 3934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "33e9aa22a4204b389c97f7f4bbbc8b11"
format_stamp: "Formatted at 2026-09-29 00:57:01 on dist-test-slave-30s6"
server_key: "5be3ad2a4c82bded9762a6e0963125ab"
server_key_iv: "cebeba361485c41739fc2f1ddd8b1aa6"
server_key_version: "encryptionkey@0"
I20260929 00:57:01.848623 3934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:01.867686 3934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:01.870656 3934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:01.870860 3934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:01.878051 3934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36415
I20260929 00:57:01.878073 4001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36415 every 8 connection(s)
I20260929 00:57:01.879072 3934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:01.881847 4002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:01.884891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 3934
I20260929 00:57:01.885012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:01.885288 392 external_mini_cluster.cc:1428] Setting key 71c9870066a897c7bd488ccabc1b0f81
I20260929 00:57:01.887822 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11: Bootstrap starting.
I20260929 00:57:01.890452 4002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:01.891382 4002 log.cc:824] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:01.893613 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11: No bootstrap required, opened a new log
Sep 29 00:57:01 dist-test-slave-30s6 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:01.896762 4002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9aa22a4204b389c97f7f4bbbc8b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36415 } }
I20260929 00:57:01.897033 4002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:01.897111 4002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e9aa22a4204b389c97f7f4bbbc8b11, State: Initialized, Role: FOLLOWER
I20260929 00:57:01.897552 4002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9aa22a4204b389c97f7f4bbbc8b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36415 } }
I20260929 00:57:01.897699 4002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:01.897789 4002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:01.897905 4002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:01.898996 4002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9aa22a4204b389c97f7f4bbbc8b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36415 } }
I20260929 00:57:01.899329 4002 leader_election.cc:302] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33e9aa22a4204b389c97f7f4bbbc8b11; no voters:
I20260929 00:57:01.899649 4002 leader_election.cc:288] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:01.899998 4007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:01.900259 4007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [term 1 LEADER]: Becoming Leader. State: Replica: 33e9aa22a4204b389c97f7f4bbbc8b11, State: Running, Role: LEADER
I20260929 00:57:01.900569 4007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9aa22a4204b389c97f7f4bbbc8b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36415 } }
I20260929 00:57:01.900986 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:01.886714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.0.98.62:36415)
0929 00:57:01.887145 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:01.887161 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:01.887195 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:57:01.887803 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:01.887819 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:01.888226 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:01.888570 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:01.889491 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:01.890572 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:01.891744 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:01.892373 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:01.892492 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:01.895224 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:01.895229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:01.895242 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:01.895267 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:01.897663 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:01.898256 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:01.898261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:01.898273 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:01.898386 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:01.898663 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:01.898665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:01.898666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:01.899100 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:01.899295 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:01.899633 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 00:57:01.899970 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":152,"threads_started":1}
I20260929 00:57:01.901048 4002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:01.901685 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e9aa22a4204b389c97f7f4bbbc8b11. Latest consensus state: current_term: 1 leader_uuid: "33e9aa22a4204b389c97f7f4bbbc8b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9aa22a4204b389c97f7f4bbbc8b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36415 } } }
I20260929 00:57:01.901803 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:01.902246 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33e9aa22a4204b389c97f7f4bbbc8b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9aa22a4204b389c97f7f4bbbc8b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36415 } } }
I20260929 00:57:01.902396 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:01.902256 4015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:01.905637 4015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:01.911319 4015 catalog_manager.cc:1347] Generated new cluster ID: 845dc79f1b714d2dbba01ff610b5e15f
I20260929 00:57:01.911423 4015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:01.942723 4015 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:01.943781 4015 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:01.947696 4015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33e9aa22a4204b389c97f7f4bbbc8b11: Generated new TSK 0
I20260929 00:57:01.948417 4015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:57:01.986368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36415
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:02.099093 4028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:02.099391 4028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:02.099473 4028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:02.104010 4028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:02.104079 4028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:02.104171 4028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:02.109721 4028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:02.110973 4028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:02.112080 4028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:02.122393 4034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:02.122393 4033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:02.122501 4036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:02.122668 4028 server_base.cc:1034] running on GCE node
I20260929 00:57:02.123190 4028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:02.123857 4028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:02.125021 4028 hybrid_clock.cc:648] HybridClock initialized: now 1790643422125009 us; error 29 us; skew 500 ppm
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:02.127836 4028 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:57:02.128777 4028 webserver.cc:460] Webserver started at http://127.0.98.1:43041/ using document root <none> and password file <none>
I20260929 00:57:02.129365 4028 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:02.129458 4028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:02.129686 4028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:02.131767 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "21c07b9fbf2942f580d7758e23b179b1"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "7b5a8d4ce453c9ad07714f0911759e06"
server_key_iv: "f6f99b51a0b72e8514e7e771c9ba5a07"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.132373 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "21c07b9fbf2942f580d7758e23b179b1"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "7b5a8d4ce453c9ad07714f0911759e06"
server_key_iv: "f6f99b51a0b72e8514e7e771c9ba5a07"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.136035 4028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:57:02.138437 4043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.139647 4028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:02.139783 4028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "21c07b9fbf2942f580d7758e23b179b1"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "7b5a8d4ce453c9ad07714f0911759e06"
server_key_iv: "f6f99b51a0b72e8514e7e771c9ba5a07"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.139895 4028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:02.156975 4028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:02.160346 4028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:02.160570 4028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:02.161175 4028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:02.162091 4028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:02.162163 4028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.162245 4028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:02.162276 4028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.172612 4028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45547
I20260929 00:57:02.172698 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45547 every 8 connection(s)
I20260929 00:57:02.173555 4028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:02.183101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 4028
I20260929 00:57:02.183240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:02.183561 392 external_mini_cluster.cc:1428] Setting key 5170a766ce79e3872d5b65233b5fb42c
I20260929 00:57:02.188072 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:02.175474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57639 (local address 127.0.98.62:36415)
0929 00:57:02.175617 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:02.175621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:02.176489 (+ 868us) server_negotiation.cc:407] Connection header received
0929 00:57:02.177868 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:02.177871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:02.177927 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:02.178034 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:02.179239 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.179770 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.180934 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.181169 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.181282 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:02.184021 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:02.184027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:02.184034 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:02.184078 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:02.185879 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.186454 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.186461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.186466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.186563 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.186932 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.186936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.186938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.187121 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:02.187228 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:02.187741 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 00:57:02.187900 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:57:02.188894 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:02.175853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36415 (local address 127.0.98.1:57639)
0929 00:57:02.176308 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:02.176372 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:02.177655 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:02.178262 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:02.178270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:02.178624 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:02.179093 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:02.179105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.179925 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.179928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:02.180809 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:02.180819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.181328 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.181333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:02.181502 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:02.182071 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:02.182086 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:02.183850 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:02.186000 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:02.186005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:02.186017 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:02.186358 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:02.186655 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:02.186657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:02.186659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:02.186823 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:02.187228 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:02.187236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:02.187501 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 00:57:02.187690 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 00:57:02.188045 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
I20260929 00:57:02.190281 4157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36415
I20260929 00:57:02.190527 4157 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:02.191006 4157 heartbeater.cc:511] Master 127.0.98.62:36415 requested a full tablet report, sending...
I20260929 00:57:02.192680 3966 ts_manager.cc:194] Registered new tserver with Master: 21c07b9fbf2942f580d7758e23b179b1 (127.0.98.1:45547)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:57:02.194350 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:57:02.212625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36415
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:02.322012 4162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:02.322376 4162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:02.322474 4162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:02.326591 4162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:02.326673 4162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:02.326762 4162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:02.332195 4162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:02.333266 4162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:02.334326 4162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:02.344010 4170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:02.343991 4168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:02.343991 4162 server_base.cc:1034] running on GCE node
W20260929 00:57:02.344056 4167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:02.344744 4162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:02.345347 4162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:02.346524 4162 hybrid_clock.cc:648] HybridClock initialized: now 1790643422346505 us; error 26 us; skew 500 ppm
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:02.349224 4162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:57:02.350068 4162 webserver.cc:460] Webserver started at http://127.0.98.2:33285/ using document root <none> and password file <none>
I20260929 00:57:02.350683 4162 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:02.350772 4162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:02.350997 4162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:02.353053 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "9216387ac286453d92870a0d27b5748c"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "5a1104bf51745a7cff9054275e33fe67"
server_key_iv: "7807a4f5fd959dbd6f8aa72e979c8e65"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.353634 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9216387ac286453d92870a0d27b5748c"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "5a1104bf51745a7cff9054275e33fe67"
server_key_iv: "7807a4f5fd959dbd6f8aa72e979c8e65"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.357373 4162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 00:57:02.359869 4177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.361038 4162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:57:02.361186 4162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "9216387ac286453d92870a0d27b5748c"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "5a1104bf51745a7cff9054275e33fe67"
server_key_iv: "7807a4f5fd959dbd6f8aa72e979c8e65"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.361297 4162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:02.375977 4162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:02.378995 4162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:02.379196 4162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:02.379777 4162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:02.380578 4162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:02.380648 4162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.380719 4162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:02.380764 4162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.389760 4162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42729
I20260929 00:57:02.389781 4290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42729 every 8 connection(s)
I20260929 00:57:02.390654 4162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:02.398870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 4162
I20260929 00:57:02.399009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:02.399343 392 external_mini_cluster.cc:1428] Setting key 703b2e957b5e7056d5ba7e0d7419d44d
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:02.405835 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:02.392531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41961 (local address 127.0.98.62:36415)
0929 00:57:02.392658 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:02.392664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:02.393497 (+ 833us) server_negotiation.cc:407] Connection header received
0929 00:57:02.394854 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:02.394857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:02.394903 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:02.394978 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:02.396121 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.396638 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.397829 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.398043 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.398124 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:02.401117 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:02.401119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:02.401122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:02.401144 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:02.403046 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.403573 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.403578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.403582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.403714 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.404231 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.404237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.404241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.404679 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:02.404837 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:02.405538 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 00:57:02.405663 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:57:02.406754 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:02.392912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36415 (local address 127.0.98.2:41961)
0929 00:57:02.393339 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:02.393379 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:02.394655 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:02.395140 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:02.395148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:02.395521 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:02.395995 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:02.396007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.396775 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.396779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:02.397704 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:02.397714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.398174 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.398255 (+ 81us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:02.398434 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:02.399128 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:02.399150 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:02.400958 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:02.403160 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:02.403166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:02.403179 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:02.403446 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:02.403863 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:02.403869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:02.403873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:02.404096 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:02.404921 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:02.404935 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:02.405226 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 00:57:02.405484 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 00:57:02.405825 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260929 00:57:02.407951 4291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36415
I20260929 00:57:02.408231 4291 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:02.408883 4291 heartbeater.cc:511] Master 127.0.98.62:36415 requested a full tablet report, sending...
I20260929 00:57:02.410075 3966 ts_manager.cc:194] Registered new tserver with Master: 9216387ac286453d92870a0d27b5748c (127.0.98.2:42729)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:57:02.410912 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:57:02.429203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36415
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:02.539675 4296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:02.540058 4296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:02.540122 4296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:02.544369 4296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:02.544430 4296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:02.544492 4296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:02.549671 4296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:02.550727 4296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:02.551728 4296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:02.561275 4302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:02.561312 4301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:02.561293 4304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:02.561460 4296 server_base.cc:1034] running on GCE node
I20260929 00:57:02.561995 4296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:02.562608 4296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:02.563763 4296 hybrid_clock.cc:648] HybridClock initialized: now 1790643422563739 us; error 39 us; skew 500 ppm
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:02.566453 4296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:57:02.567298 4296 webserver.cc:460] Webserver started at http://127.0.98.3:40909/ using document root <none> and password file <none>
I20260929 00:57:02.567854 4296 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:02.567927 4296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:02.568158 4296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:02.570065 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "f62d3ee502d148b284e717ee93fedd66"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "d5417dd34e03f6f63a8f883a09d0c159"
server_key_iv: "5da51dbaa755d55771f8f480670f17a4"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.570669 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f62d3ee502d148b284e717ee93fedd66"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "d5417dd34e03f6f63a8f883a09d0c159"
server_key_iv: "5da51dbaa755d55771f8f480670f17a4"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.574384 4296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:02.576676 4311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.577847 4296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:02.577968 4296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "f62d3ee502d148b284e717ee93fedd66"
format_stamp: "Formatted at 2026-09-29 00:57:02 on dist-test-slave-30s6"
server_key: "d5417dd34e03f6f63a8f883a09d0c159"
server_key_iv: "5da51dbaa755d55771f8f480670f17a4"
server_key_version: "encryptionkey@0"
I20260929 00:57:02.578069 4296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:02.594937 4296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:02.597726 4296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:02.597939 4296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:02.598587 4296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:02.599500 4296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:02.599548 4296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.599612 4296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:02.599651 4296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:02.608603 4296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39315
I20260929 00:57:02.608635 4424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39315 every 8 connection(s)
I20260929 00:57:02.609508 4296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:02.615236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 4296
I20260929 00:57:02.615391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:02.615705 392 external_mini_cluster.cc:1428] Setting key ff6b57f96429dcdc10a5a21023faeb73
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:02.623114 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:02.611322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36425 (local address 127.0.98.62:36415)
0929 00:57:02.611452 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:02.611456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:02.612269 (+ 813us) server_negotiation.cc:407] Connection header received
0929 00:57:02.613574 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:02.613577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:02.613621 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:02.613694 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:02.614913 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.615499 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.617077 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.617389 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.617492 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:02.620008 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:02.620012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:02.620014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:02.620037 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:02.621239 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.621695 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.621697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.621699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.621766 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.622044 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.622046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.622047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.622379 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:02.622457 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:02.622908 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 00:57:02.622999 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:57:02.624032 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:02.611697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36415 (local address 127.0.98.3:36425)
0929 00:57:02.612115 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:02.612156 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:02.613385 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:02.613854 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:02.613861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:02.614307 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:02.614786 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:02.614801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.615630 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.615634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:02.616931 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:02.616945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.617526 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.617529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:02.617737 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:02.618441 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:02.618458 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:02.619857 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:02.621351 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:02.621356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:02.621368 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:02.621612 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:02.621840 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:02.621842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:02.621844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:02.621970 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:02.622468 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:02.622473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:02.622704 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 00:57:02.622861 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 00:57:02.623111 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260929 00:57:02.625198 4425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36415
I20260929 00:57:02.625451 4425 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:02.625932 4425 heartbeater.cc:511] Master 127.0.98.62:36415 requested a full tablet report, sending...
I20260929 00:57:02.626976 3966 ts_manager.cc:194] Registered new tserver with Master: f62d3ee502d148b284e717ee93fedd66 (127.0.98.3:39315)
I20260929 00:57:02.627530 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36425
I20260929 00:57:02.629397 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:57:02.654409 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:02.645172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.0.98.1:45547)
0929 00:57:02.645478 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:02.645492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:02.645516 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:57:02.645564 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:02.645568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:02.645752 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:02.645874 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:02.646571 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.647343 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.648126 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.648500 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.648579 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:02.650957 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:02.650966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:02.650978 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:02.651001 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:02.652803 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.653252 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.653257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.653259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.653351 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.653554 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.653556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.653557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.653767 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:02.653931 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:02.654065 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 00:57:02.654254 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":120,"threads_started":1}
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:02.664429 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:02.657179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.0.98.62:36415)
0929 00:57:02.657287 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:02.657290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:02.657303 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:57:02.657368 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:02.657370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:02.657407 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:02.657469 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:02.657959 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.658544 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.659245 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:02.659452 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:02.659522 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:02.661609 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:02.661612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:02.661614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:02.661635 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:02.663066 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.663414 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.663416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.663417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.663485 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:02.663692 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:02.663694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:02.663695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:02.663891 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:02.663989 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:02.664222 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 00:57:02.664316 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 00:57:02.664831 3966 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49132
I20260929 00:57:02.665480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 4028
I20260929 00:57:02.671021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 4162
I20260929 00:57:02.676220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 4296
I20260929 00:57:02.681454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 3934
2026-09-29T00:57:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2148 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[4437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[4437](info): set up 2 sockets
Sep 29 00:57:02 dist-test-slave-30s6 krb5kdc[4437](info): commencing operation
krb5kdc: starting...
W20260929 00:57:03.734323 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:03 dist-test-slave-30s6 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:57:03.857636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:03.968135 4450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:03.968466 4450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:03.968552 4450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:03.972724 4450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:03.972821 4450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:03.972859 4450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:03.972918 4450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:03.972959 4450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:03.979782 4450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:03.981367 4450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:03.982788 4450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:03.993191 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:03.993345 4450 server_base.cc:1034] running on GCE node
W20260929 00:57:03.993191 4456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:03.993311 4458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:03.994313 4450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:03.995843 4450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:03.997004 4450 hybrid_clock.cc:648] HybridClock initialized: now 1790643423996973 us; error 46 us; skew 500 ppm
Sep 29 00:57:03 dist-test-slave-30s6 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:04.000767 4450 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:57:04.001734 4450 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:04.001829 4450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:04.002123 4450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:04.004604 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "bc42759008b9433081d7dd80926b6c39"
format_stamp: "Formatted at 2026-09-29 00:57:03 on dist-test-slave-30s6"
server_key: "d3b48393acac70c3bc4692249af8d44d"
server_key_iv: "427a5d0dd50caf1941e277c2147b0a80"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.005290 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc42759008b9433081d7dd80926b6c39"
format_stamp: "Formatted at 2026-09-29 00:57:03 on dist-test-slave-30s6"
server_key: "d3b48393acac70c3bc4692249af8d44d"
server_key_iv: "427a5d0dd50caf1941e277c2147b0a80"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.010548 4450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 00:57:04.013530 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.015164 4450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:04.015285 4450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "bc42759008b9433081d7dd80926b6c39"
format_stamp: "Formatted at 2026-09-29 00:57:03 on dist-test-slave-30s6"
server_key: "d3b48393acac70c3bc4692249af8d44d"
server_key_iv: "427a5d0dd50caf1941e277c2147b0a80"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.015393 4450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:04.035125 4450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:04.038390 4450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:04.038546 4450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:04.045553 4450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33933
I20260929 00:57:04.045912 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33933 every 8 connection(s)
I20260929 00:57:04.046332 4450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:04.049121 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:04.054093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 4450
I20260929 00:57:04.054268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:04.054519 392 external_mini_cluster.cc:1428] Setting key f99ea9b986865ae9966cb80eb0d2fe67
I20260929 00:57:04.054817 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39: Bootstrap starting.
I20260929 00:57:04.057546 4517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:04.058475 4517 log.cc:824] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:04.060719 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39: No bootstrap required, opened a new log
Sep 29 00:57:04 dist-test-slave-30s6 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:04.065224 4517 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc42759008b9433081d7dd80926b6c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } }
I20260929 00:57:04.065493 4517 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:04.065585 4517 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc42759008b9433081d7dd80926b6c39, State: Initialized, Role: FOLLOWER
I20260929 00:57:04.066131 4517 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc42759008b9433081d7dd80926b6c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } }
I20260929 00:57:04.066334 4517 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:04.066406 4517 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:04.066507 4517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:04.067678 4517 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc42759008b9433081d7dd80926b6c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } }
I20260929 00:57:04.068037 4517 leader_election.cc:302] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc42759008b9433081d7dd80926b6c39; no voters:
I20260929 00:57:04.068397 4517 leader_election.cc:288] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:04.068537 4522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:04.068830 4522 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [term 1 LEADER]: Becoming Leader. State: Replica: bc42759008b9433081d7dd80926b6c39, State: Running, Role: LEADER
I20260929 00:57:04.069213 4522 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc42759008b9433081d7dd80926b6c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } }
I20260929 00:57:04.069770 4517 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:04.070519 4522 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc42759008b9433081d7dd80926b6c39. Latest consensus state: current_term: 1 leader_uuid: "bc42759008b9433081d7dd80926b6c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc42759008b9433081d7dd80926b6c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } } }
I20260929 00:57:04.070673 4522 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:04.070503 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc42759008b9433081d7dd80926b6c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc42759008b9433081d7dd80926b6c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } } }
I20260929 00:57:04.070894 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:04.071178 4531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:04.071319 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.055968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.0.98.62:33933)
0929 00:57:04.056382 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:04.056401 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:04.056443 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:57:04.057294 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:04.057317 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:04.057756 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:04.058151 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:04.059046 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.060424 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.061330 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.061995 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.062209 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.064975 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:04.064981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:04.064998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:04.065030 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:04.067182 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.067605 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.067612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.067620 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.067786 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.068159 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.068162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.068164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.068701 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:04.068908 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:04.069748 (+ 840us) server_negotiation.cc:299] Negotiation successful
0929 00:57:04.070072 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":100,"threads_started":1}
I20260929 00:57:04.075042 4531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:04.081588 4531 catalog_manager.cc:1347] Generated new cluster ID: 98165430cfb74add8671d7f7596c773f
I20260929 00:57:04.081686 4531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:04.097128 4531 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:04.098345 4531 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:04.102414 4531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39: Generated new TSK 0
I20260929 00:57:04.103147 4531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:57:04.135146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:04.250905 4543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:04.251214 4543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:04.251264 4543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:04.255705 4543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:04.255793 4543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:04.255859 4543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:04.261811 4543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:04.263024 4543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:04.264138 4543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:04.274518 4549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:04.274555 4551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:04.274535 4548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:04.274686 4543 server_base.cc:1034] running on GCE node
I20260929 00:57:04.275348 4543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:04.276134 4543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:04.277325 4543 hybrid_clock.cc:648] HybridClock initialized: now 1790643424277301 us; error 41 us; skew 500 ppm
Sep 29 00:57:04 dist-test-slave-30s6 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:04.280563 4543 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:57:04.281431 4543 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:04.281534 4543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:04.281798 4543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:04.284031 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "3f0ce07243fcc07374cbe39776f3392b"
server_key_iv: "5b49feb97980884c4793f0026cfad2c8"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.284631 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "3f0ce07243fcc07374cbe39776f3392b"
server_key_iv: "5b49feb97980884c4793f0026cfad2c8"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.288609 4543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 00:57:04.291424 4557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.292989 4543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:04.293156 4543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "3f0ce07243fcc07374cbe39776f3392b"
server_key_iv: "5b49feb97980884c4793f0026cfad2c8"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.293286 4543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:04.306371 4543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:04.310710 4543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:04.310889 4543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:04.311460 4543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:04.312279 4543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:04.312327 4543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.312366 4543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:04.312381 4543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.321442 4543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44467
I20260929 00:57:04.321465 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44467 every 8 connection(s)
I20260929 00:57:04.322167 4543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:04.332094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 4543
I20260929 00:57:04.332244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:04.332600 392 external_mini_cluster.cc:1428] Setting key 1526ca5869d6ea595ee1c9bd5cd91301
Sep 29 00:57:04 dist-test-slave-30s6 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:04.338980 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.324486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53913 (local address 127.0.98.62:33933)
0929 00:57:04.324665 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:04.324670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:04.325605 (+ 935us) server_negotiation.cc:407] Connection header received
0929 00:57:04.327134 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:04.327137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:04.327189 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:04.327272 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:04.328636 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.329361 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.330945 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.331233 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.331331 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.334594 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:04.334599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:04.334601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:04.334631 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:04.336467 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.337038 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.337042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.337044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.337125 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.337600 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.337603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.337604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.337822 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:04.337919 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:04.338709 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 00:57:04.338847 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 00:57:04.340279 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.324950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33933 (local address 127.0.98.1:53913)
0929 00:57:04.325444 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:04.325490 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:04.326887 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:04.327475 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:04.327483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:04.327923 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:04.328440 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.328454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.329579 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.329582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.330799 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.330815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.331401 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.331405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.331614 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.332407 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:04.332425 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:04.334418 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:04.336599 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:04.336605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:04.336626 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:04.336921 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:04.337229 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:04.337235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:04.337240 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:04.337441 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:04.337943 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:04.337954 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:04.338350 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 00:57:04.338578 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 00:57:04.339031 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260929 00:57:04.341879 4671 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33933
I20260929 00:57:04.342171 4671 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:04.342782 4671 heartbeater.cc:511] Master 127.0.98.62:33933 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:57:04.344506 4479 ts_manager.cc:194] Registered new tserver with Master: 5e79d6c8e0ba4b208e4b891ea58ed3be (127.0.98.1:44467)
I20260929 00:57:04.345733 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:57:04.366273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:04.480887 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:04.481200 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:04.481281 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:04.485386 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:04.485483 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:04.485579 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:04.491068 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:04.492231 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:04.493311 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:04.503139 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:04.503165 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:04.503068 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:04.503520 4676 server_base.cc:1034] running on GCE node
I20260929 00:57:04.503985 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:04.504726 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:04.505955 4676 hybrid_clock.cc:648] HybridClock initialized: now 1790643424505921 us; error 49 us; skew 500 ppm
Sep 29 00:57:04 dist-test-slave-30s6 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:04.509137 4676 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:57:04.509945 4676 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:04.510056 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:04.510370 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:04.512537 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "e7710da8aec1438eace0473955f8f5b2"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "82a372282045efa1acaf412a0341d280"
server_key_iv: "cb850cb30c72a4ab55731026c54d9295"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.513149 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e7710da8aec1438eace0473955f8f5b2"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "82a372282045efa1acaf412a0341d280"
server_key_iv: "cb850cb30c72a4ab55731026c54d9295"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.517634 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 00:57:04.520325 4690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.521652 4676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 00:57:04.521798 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "e7710da8aec1438eace0473955f8f5b2"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "82a372282045efa1acaf412a0341d280"
server_key_iv: "cb850cb30c72a4ab55731026c54d9295"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.521921 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:04.545218 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:04.548832 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:04.549063 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:04.549688 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:04.550577 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:04.550653 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.550734 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:04.550781 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.559842 4676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43673
I20260929 00:57:04.559859 4803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43673 every 8 connection(s)
I20260929 00:57:04.560487 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:04.563099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 4676
I20260929 00:57:04.563235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:04.563584 392 external_mini_cluster.cc:1428] Setting key a88958020a6fc58b86856b00296bf8aa
Sep 29 00:57:04 dist-test-slave-30s6 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:57:04.576680 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.562323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57549 (local address 127.0.98.62:33933)
0929 00:57:04.562471 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:04.562476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:04.563537 (+ 1061us) server_negotiation.cc:407] Connection header received
0929 00:57:04.565015 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:04.565019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:04.565080 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:04.565148 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:04.566436 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.567242 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.569099 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.569315 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.569486 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.572634 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:04.572637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:04.572640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:04.572674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:04.574327 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.574949 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.574953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.574955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.575040 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.575432 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.575435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.575437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.575638 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:04.575780 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:04.576324 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 00:57:04.576452 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:57:04.577512 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.562766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33933 (local address 127.0.98.2:57549)
0929 00:57:04.563339 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:04.563404 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:04.564803 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:04.565348 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:04.565356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:04.565721 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:04.566273 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.566289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.567378 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.567385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.568935 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.568949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.569531 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.569538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.569782 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.570581 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:04.570598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:04.572439 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:04.574460 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:04.574466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:04.574484 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:04.574851 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:04.575144 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:04.575147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:04.575149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:04.575302 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:04.575750 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:04.575758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:04.576080 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 00:57:04.576274 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 00:57:04.576629 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":105,"threads_started":1}
I20260929 00:57:04.578770 4804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33933
I20260929 00:57:04.579035 4804 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:04.579525 4804 heartbeater.cc:511] Master 127.0.98.62:33933 requested a full tablet report, sending...
I20260929 00:57:04.580614 4479 ts_manager.cc:194] Registered new tserver with Master: e7710da8aec1438eace0473955f8f5b2 (127.0.98.2:43673)
I20260929 00:57:04.581462 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:57:04.593277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:04.705039 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:04.705325 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:04.705406 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:04.709585 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:04.709651 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:04.709750 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:04.715219 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:04.716305 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:04.717363 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:04.726843 4815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:04.726843 4814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:04.726842 4817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:04.727244 4809 server_base.cc:1034] running on GCE node
I20260929 00:57:04.727686 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:04.728353 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:04.729540 4809 hybrid_clock.cc:648] HybridClock initialized: now 1790643424729529 us; error 36 us; skew 500 ppm
Sep 29 00:57:04 dist-test-slave-30s6 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:04.732331 4809 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:57:04.733088 4809 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:04.733188 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:04.733418 4809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:04.735549 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c440cfce80341b1ba00c055c6213794"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "7f98df5cea76a317b549f6b50a39b0b1"
server_key_iv: "d1e4ad74285559311e746b386f0c075d"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.736111 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c440cfce80341b1ba00c055c6213794"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "7f98df5cea76a317b549f6b50a39b0b1"
server_key_iv: "d1e4ad74285559311e746b386f0c075d"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.740181 4809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 00:57:04.742700 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.744019 4809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:04.744189 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "6c440cfce80341b1ba00c055c6213794"
format_stamp: "Formatted at 2026-09-29 00:57:04 on dist-test-slave-30s6"
server_key: "7f98df5cea76a317b549f6b50a39b0b1"
server_key_iv: "d1e4ad74285559311e746b386f0c075d"
server_key_version: "encryptionkey@0"
I20260929 00:57:04.744310 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:04.794545 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:04.797683 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:04.797888 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:04.798563 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:04.799422 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:04.799494 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.799566 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:04.799620 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:04.808475 4809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33933
I20260929 00:57:04.808490 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33933 every 8 connection(s)
I20260929 00:57:04.809072 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:04.810274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 4809
I20260929 00:57:04.810408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:04.810696 392 external_mini_cluster.cc:1428] Setting key 55b2f576c05c893d9f63dc9f20139a9b
Sep 29 00:57:04 dist-test-slave-30s6 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:04.822728 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.810928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46197 (local address 127.0.98.62:33933)
0929 00:57:04.811081 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:04.811086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:04.812102 (+ 1016us) server_negotiation.cc:407] Connection header received
0929 00:57:04.813618 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:04.813620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:04.813680 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:04.813746 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:04.814929 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.815437 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.816513 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.816730 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.816808 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.819187 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:04.819190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:04.819192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:04.819213 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:04.820587 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.821121 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.821127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.821129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.821212 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.821548 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.821553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.821555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.821742 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:04.821852 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:04.822406 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 00:57:04.822525 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:57:04.823675 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.811430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33933 (local address 127.0.98.3:46197)
0929 00:57:04.811925 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:04.811997 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:04.813418 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:04.813882 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:04.813888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:04.814322 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:04.814813 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.814826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.815537 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.815547 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.816405 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.816413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.816853 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.816855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.816990 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.817591 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:04.817607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:04.819044 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:04.820711 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:04.820718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:04.820733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:04.821015 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:04.821306 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:04.821310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:04.821312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:04.821452 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:04.821849 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:04.821856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:04.822108 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 00:57:04.822354 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 00:57:04.822699 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260929 00:57:04.824834 4937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33933
I20260929 00:57:04.825075 4937 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:04.825546 4937 heartbeater.cc:511] Master 127.0.98.62:33933 requested a full tablet report, sending...
I20260929 00:57:04.826576 4481 ts_manager.cc:194] Registered new tserver with Master: 6c440cfce80341b1ba00c055c6213794 (127.0.98.3:33933)
I20260929 00:57:04.827157 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46197
I20260929 00:57:04.836755 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:57:04.845808 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.838789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.0.98.62:33933)
0929 00:57:04.838970 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:04.838974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:04.839034 (+ 60us) server_negotiation.cc:407] Connection header received
0929 00:57:04.839157 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:04.839159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:04.839200 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:04.839266 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:04.839883 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.840407 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.841220 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.841472 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.841567 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.842920 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:04.842923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:04.842924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:04.842944 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:04.844300 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.844671 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.844675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.844676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.844749 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:04.844969 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:04.844971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:04.844972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:04.845204 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:04.845306 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:04.845494 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 00:57:04.845586 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 00:57:04.848785 4481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:04.851796 4481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:04.866233 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.860166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.0.98.2:43673)
0929 00:57:04.860758 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:04.860772 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:04.861397 (+ 625us) server_negotiation.cc:407] Connection header received
0929 00:57:04.861492 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:04.861501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:04.861716 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:04.861887 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:04.862227 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.863410 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.865037 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.865563 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.865695 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.865839 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:04.865965 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 00:57:04.866055 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":90,"threads_started":1}
I20260929 00:57:04.866226 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.860044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43673 (local address 127.0.0.1:54418)
0929 00:57:04.861084 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:04.861116 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:04.861248 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:04.861887 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:04.861893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:04.861915 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:04.862070 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.862078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.863584 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.863588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.864900 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.864910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.865691 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.865695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.865794 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.865828 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:57:04.865929 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:57:04.866043 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":52,"thread_start_us":92,"threads_started":1}
I20260929 00:57:04.866533 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.860729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44467 (local address 127.0.0.1:33348)
0929 00:57:04.861224 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:04.861237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:04.861324 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:04.862071 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:04.862074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:04.862105 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:04.862315 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.862322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.863654 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.863656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.864918 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.864924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.866058 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.866062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.866169 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.866243 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 00:57:04.866315 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 00:57:04.866384 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":106,"spinlock_wait_cycles":1792,"thread_start_us":93,"threads_started":1}
I20260929 00:57:04.866560 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.860299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.0.98.3:33933)
0929 00:57:04.860912 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:04.860926 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:04.861557 (+ 631us) server_negotiation.cc:407] Connection header received
0929 00:57:04.861669 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:04.861674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:04.861812 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:04.861935 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:04.862438 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.863189 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.864809 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.865622 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.865744 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.865901 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:04.866291 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 00:57:04.866391 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":99,"threads_started":1}
I20260929 00:57:04.867306 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.861110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.0.98.1:44467)
0929 00:57:04.861458 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:04.861466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:04.861493 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:57:04.861584 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:04.861589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:04.861790 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:04.861935 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:04.862448 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.863534 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.865029 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.865764 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.865864 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.866001 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:04.867026 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0929 00:57:04.867146 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":70,"threads_started":1}
I20260929 00:57:04.867535 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:04.860072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33933 (local address 127.0.0.1:47868)
0929 00:57:04.861001 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:04.861443 (+ 442us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:04.861560 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:04.862074 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:04.862077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:04.862114 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:04.862304 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.862310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.863331 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.863338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.864679 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:04.864689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:04.865970 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:04.865974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:04.866088 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:04.866114 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 00:57:04.867211 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0929 00:57:04.867280 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":93,"spinlock_wait_cycles":18688,"thread_start_us":124,"threads_started":1}
I20260929 00:57:04.869299 4738 tablet_service.cc:1467] Processing CreateTablet for tablet c0994752f27d4173b4842d60c8f73cc7 (DEFAULT_TABLE table=test-table [id=54d2438879d645aca085c1788a5d4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:04.870517 4738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0994752f27d4173b4842d60c8f73cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:04.870376 4871 tablet_service.cc:1467] Processing CreateTablet for tablet c0994752f27d4173b4842d60c8f73cc7 (DEFAULT_TABLE table=test-table [id=54d2438879d645aca085c1788a5d4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:04.871475 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0994752f27d4173b4842d60c8f73cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:04.871783 4605 tablet_service.cc:1467] Processing CreateTablet for tablet c0994752f27d4173b4842d60c8f73cc7 (DEFAULT_TABLE table=test-table [id=54d2438879d645aca085c1788a5d4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:04.872694 4605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0994752f27d4173b4842d60c8f73cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:04.877930 4954 tablet_bootstrap.cc:492] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be: Bootstrap starting.
I20260929 00:57:04.878876 4956 tablet_bootstrap.cc:492] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Bootstrap starting.
I20260929 00:57:04.880518 4954 tablet_bootstrap.cc:654] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:04.881090 4956 tablet_bootstrap.cc:654] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:04.881433 4954 log.cc:824] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:04.882254 4955 tablet_bootstrap.cc:492] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: Bootstrap starting.
I20260929 00:57:04.882417 4956 log.cc:824] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:04.883682 4954 tablet_bootstrap.cc:492] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be: No bootstrap required, opened a new log
I20260929 00:57:04.883870 4954 ts_tablet_manager.cc:1397] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:57:04.884346 4955 tablet_bootstrap.cc:654] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:04.884362 4956 tablet_bootstrap.cc:492] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: No bootstrap required, opened a new log
I20260929 00:57:04.884548 4956 ts_tablet_manager.cc:1397] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:57:04.885581 4955 log.cc:824] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:04.887578 4955 tablet_bootstrap.cc:492] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: No bootstrap required, opened a new log
I20260929 00:57:04.887534 4954 raft_consensus.cc:348] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:04.887785 4955 ts_tablet_manager.cc:1397] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:57:04.887835 4954 raft_consensus.cc:374] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:04.887897 4954 raft_consensus.cc:729] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e79d6c8e0ba4b208e4b891ea58ed3be, State: Initialized, Role: FOLLOWER
I20260929 00:57:04.888350 4954 consensus_queue.cc:260] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:04.888494 4956 raft_consensus.cc:348] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:04.888777 4956 raft_consensus.cc:374] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:04.888849 4956 raft_consensus.cc:729] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c440cfce80341b1ba00c055c6213794, State: Initialized, Role: FOLLOWER
I20260929 00:57:04.889379 4954 ts_tablet_manager.cc:1428] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 00:57:04.889324 4956 consensus_queue.cc:260] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:04.889856 4671 heartbeater.cc:503] Master 127.0.98.62:33933 was elected leader, sending a full tablet report...
I20260929 00:57:04.890254 4937 heartbeater.cc:503] Master 127.0.98.62:33933 was elected leader, sending a full tablet report...
I20260929 00:57:04.890307 4956 ts_tablet_manager.cc:1428] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 00:57:04.891759 4955 raft_consensus.cc:348] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:04.892041 4955 raft_consensus.cc:374] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:04.892114 4955 raft_consensus.cc:729] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7710da8aec1438eace0473955f8f5b2, State: Initialized, Role: FOLLOWER
I20260929 00:57:04.892584 4955 consensus_queue.cc:260] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:04.893378 4804 heartbeater.cc:503] Master 127.0.98.62:33933 was elected leader, sending a full tablet report...
I20260929 00:57:04.893603 4955 ts_tablet_manager.cc:1428] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260929 00:57:05.012276 4962 raft_consensus.cc:482] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:05.012535 4962 raft_consensus.cc:504] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:05.013805 4962 leader_election.cc:288] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c440cfce80341b1ba00c055c6213794 (127.0.98.3:33933), 5e79d6c8e0ba4b208e4b891ea58ed3be (127.0.98.1:44467)
I20260929 00:57:05.018718 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.014323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46345 (local address 127.0.98.1:44467)
0929 00:57:05.014429 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.014433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.014670 (+ 237us) server_negotiation.cc:407] Connection header received
0929 00:57:05.014726 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.014728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.014769 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.014855 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:05.015095 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.015796 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.017573 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.018089 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.018213 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:05.018280 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.018515 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.018586 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:57:05.018734 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.014127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44467 (local address 127.0.98.2:46345)
0929 00:57:05.014547 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.014565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:05.014658 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:05.014866 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:05.014871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:05.014889 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:05.014990 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:05.014994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.015924 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.015928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:05.017418 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:05.017430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.018259 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.018263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:05.018365 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:05.018396 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:57:05.018498 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 00:57:05.018581 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":109,"thread_start_us":137,"threads_started":1}
I20260929 00:57:05.018939 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.014003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33933 (local address 127.0.98.2:43625)
0929 00:57:05.014144 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.014160 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:05.014288 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:05.014588 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:05.014593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:05.014619 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:05.014761 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:05.014766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.015812 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.015816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:05.017418 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:05.017430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.018447 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.018452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:05.018592 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:05.018611 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:57:05.018712 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:57:05.018790 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":29,"spinlock_wait_cycles":2304}
I20260929 00:57:05.018936 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.014149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43625 (local address 127.0.98.3:33933)
0929 00:57:05.014319 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.014325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.014342 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:57:05.014403 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.014406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.014463 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.014541 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:05.014869 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.015674 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.017618 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.018317 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.018441 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:05.018506 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.018730 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.018809 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 00:57:05.019601 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0994752f27d4173b4842d60c8f73cc7" candidate_uuid: "e7710da8aec1438eace0473955f8f5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c440cfce80341b1ba00c055c6213794" is_pre_election: true
I20260929 00:57:05.019610 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0994752f27d4173b4842d60c8f73cc7" candidate_uuid: "e7710da8aec1438eace0473955f8f5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" is_pre_election: true
I20260929 00:57:05.019955 4891 raft_consensus.cc:2454] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7710da8aec1438eace0473955f8f5b2 in term 0.
I20260929 00:57:05.019950 4625 raft_consensus.cc:2454] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7710da8aec1438eace0473955f8f5b2 in term 0.
I20260929 00:57:05.020470 4691 leader_election.cc:302] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e79d6c8e0ba4b208e4b891ea58ed3be, e7710da8aec1438eace0473955f8f5b2; no voters:
I20260929 00:57:05.020787 4962 raft_consensus.cc:2781] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:05.020890 4962 raft_consensus.cc:482] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:05.020936 4962 raft_consensus.cc:3037] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:05.022013 4962 raft_consensus.cc:504] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:05.022436 4962 leader_election.cc:288] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6c440cfce80341b1ba00c055c6213794 (127.0.98.3:33933), 5e79d6c8e0ba4b208e4b891ea58ed3be (127.0.98.1:44467)
I20260929 00:57:05.022817 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0994752f27d4173b4842d60c8f73cc7" candidate_uuid: "e7710da8aec1438eace0473955f8f5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be"
I20260929 00:57:05.022840 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0994752f27d4173b4842d60c8f73cc7" candidate_uuid: "e7710da8aec1438eace0473955f8f5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c440cfce80341b1ba00c055c6213794"
I20260929 00:57:05.022948 4891 raft_consensus.cc:3037] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:05.022949 4625 raft_consensus.cc:3037] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:05.024224 4625 raft_consensus.cc:2454] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7710da8aec1438eace0473955f8f5b2 in term 1.
I20260929 00:57:05.024224 4891 raft_consensus.cc:2454] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7710da8aec1438eace0473955f8f5b2 in term 1.
I20260929 00:57:05.024595 4693 leader_election.cc:302] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c440cfce80341b1ba00c055c6213794, e7710da8aec1438eace0473955f8f5b2; no voters:
I20260929 00:57:05.024813 4962 raft_consensus.cc:2781] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:05.025022 4962 raft_consensus.cc:686] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 1 LEADER]: Becoming Leader. State: Replica: e7710da8aec1438eace0473955f8f5b2, State: Running, Role: LEADER
I20260929 00:57:05.025416 4962 consensus_queue.cc:237] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } }
I20260929 00:57:05.028600 4481 catalog_manager.cc:5495] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7710da8aec1438eace0473955f8f5b2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e7710da8aec1438eace0473955f8f5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7710da8aec1438eace0473955f8f5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 } health_report { overall_health: UNKNOWN } } }
W20260929 00:57:05.050518 4512 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.62:33933
W20260929 00:57:05.059999 4938 tablet.cc:2369] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:57:05.061690 4805 tablet.cc:2369] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:05.063843 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.060219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.98.2:43673)
0929 00:57:05.060374 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.060378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.060395 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:57:05.060479 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.060481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.060549 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.060671 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:05.060927 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.061428 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.062409 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:05.062642 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:05.062756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:05.063060 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:05.063455 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:05.063594 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.063652 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.063706 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:57:05.069604 4891 raft_consensus.cc:1261] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 1 FOLLOWER]: Refusing update from remote peer e7710da8aec1438eace0473955f8f5b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:05.069604 4625 raft_consensus.cc:1261] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 1 FOLLOWER]: Refusing update from remote peer e7710da8aec1438eace0473955f8f5b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:05.070418 4962 consensus_queue.cc:1035] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e79d6c8e0ba4b208e4b891ea58ed3be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:05.070634 4964 consensus_queue.cc:1035] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c440cfce80341b1ba00c055c6213794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 00:57:05.073844 4672 tablet.cc:2369] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:05.079777 4968 mvcc.cc:205] Tried to move back new op lower bound from 7334475469077934080 to 7334475468907233280. Current Snapshot: MvccSnapshot[applied={T|T < 7334475469077934080 or (T in {7334475469077934080})}]
I20260929 00:57:05.084822 4479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:05.084992 4479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:05.087414 4479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc42759008b9433081d7dd80926b6c39: Sending DeleteTablet for 3 replicas of tablet c0994752f27d4173b4842d60c8f73cc7
I20260929 00:57:05.088137 4738 tablet_service.cc:1514] Processing DeleteTablet for tablet c0994752f27d4173b4842d60c8f73cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54418
I20260929 00:57:05.088259 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet c0994752f27d4173b4842d60c8f73cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47868
I20260929 00:57:05.088299 4605 tablet_service.cc:1514] Processing DeleteTablet for tablet c0994752f27d4173b4842d60c8f73cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33348
I20260929 00:57:05.088655 4976 tablet_replica.cc:330] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: stopping tablet replica
I20260929 00:57:05.088701 4978 tablet_replica.cc:330] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be: stopping tablet replica
I20260929 00:57:05.089004 4978 raft_consensus.cc:2229] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:05.089112 4977 tablet_replica.cc:330] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: stopping tablet replica
I20260929 00:57:05.089298 4978 raft_consensus.cc:2258] T c0994752f27d4173b4842d60c8f73cc7 P 5e79d6c8e0ba4b208e4b891ea58ed3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:05.089373 4977 raft_consensus.cc:2229] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:05.089499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 4543
I20260929 00:57:05.090740 4977 raft_consensus.cc:2258] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:05.092314 4977 ts_tablet_manager.cc:1905] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:05.094893 4976 raft_consensus.cc:2229] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:05.095103 4977 ts_tablet_manager.cc:1918] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:05.095201 4977 log.cc:1196] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/c0994752f27d4173b4842d60c8f73cc7
I20260929 00:57:05.095366 4976 raft_consensus.cc:2258] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:05.095541 4977 ts_tablet_manager.cc:1939] T c0994752f27d4173b4842d60c8f73cc7 P 6c440cfce80341b1ba00c055c6213794: Deleting consensus metadata
W20260929 00:57:05.096434 4465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44467 (error 108)
I20260929 00:57:05.096902 4467 catalog_manager.cc:4841] TS 6c440cfce80341b1ba00c055c6213794 (127.0.98.3:33933): tablet c0994752f27d4173b4842d60c8f73cc7 (table test-table [id=54d2438879d645aca085c1788a5d4a43]) successfully deleted
I20260929 00:57:05.097149 4976 ts_tablet_manager.cc:1905] T c0994752f27d4173b4842d60c8f73cc7 P e7710da8aec1438eace0473955f8f5b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:05.097195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 4676
I20260929 00:57:05.099795 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.099375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44467 (local address 127.0.0.1:33352)
0929 00:57:05.099606 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.099689 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260929 00:57:05.100749 4465 catalog_manager.cc:4584] TS 5e79d6c8e0ba4b208e4b891ea58ed3be (127.0.98.1:44467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0994752f27d4173b4842d60c8f73cc7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44467: connect: Connection refused (error 111)
I20260929 00:57:05.103720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 4809
I20260929 00:57:05.104605 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.103841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43673 (local address 127.0.0.1:54446)
0929 00:57:05.104447 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.104512 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510}
I20260929 00:57:05.110419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 4450
2026-09-29T00:57:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2429 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T00:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:05Z Disabled control of system clock
I20260929 00:57:05.141741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 00:57:05.263443 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:05.263788 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:05.263834 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:05.268270 4983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:05.268350 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:05.268365 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:05.268384 4983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:05.268401 4983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:05.273370 4983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:05.274578 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:05.275971 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:05.285545 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:05.285526 4983 server_base.cc:1034] running on GCE node
W20260929 00:57:05.285501 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:05.285524 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:05.286516 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:05.287803 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:05.288987 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790643425288917 us; error 87 us; skew 500 ppm
I20260929 00:57:05.295512 4983 webserver.cc:460] Webserver started at http://127.0.98.62:44799/ using document root <none> and password file <none>
I20260929 00:57:05.296092 4983 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:05.296164 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:05.296335 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:05.298527 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "d10f932c85c64f668d72473bdfa94f10"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "f487f6cb4b99724859f26582f1fdeb1e"
server_key_iv: "e52151127b6b6b3fdc5817b9af8606aa"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.299047 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "d10f932c85c64f668d72473bdfa94f10"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "f487f6cb4b99724859f26582f1fdeb1e"
server_key_iv: "e52151127b6b6b3fdc5817b9af8606aa"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.302728 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:05.305224 4997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.306602 4983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:05.306739 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "d10f932c85c64f668d72473bdfa94f10"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "f487f6cb4b99724859f26582f1fdeb1e"
server_key_iv: "e52151127b6b6b3fdc5817b9af8606aa"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.306852 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:05.311517 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:05.312114 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:05.312299 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:05.319067 4983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41743
I20260929 00:57:05.319085 5049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41743 every 8 connection(s)
I20260929 00:57:05.319980 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:05.322999 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:05.327800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 4983
I20260929 00:57:05.327915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:05.328274 392 external_mini_cluster.cc:1428] Setting key deaddce161b3586273d84fa8dbd7c134
I20260929 00:57:05.328509 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10: Bootstrap starting.
I20260929 00:57:05.331264 5050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:05.332137 5050 log.cc:824] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:05.334100 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10: No bootstrap required, opened a new log
I20260929 00:57:05.335821 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.329766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.0.98.62:41743)
0929 00:57:05.330483 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.330504 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.330545 (+ 41us) server_negotiation.cc:407] Connection header received
0929 00:57:05.331481 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.331509 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.331983 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.332357 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:05.334520 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:05.334524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:05.334536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:05.334546 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:05.334601 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:05.334625 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:05.334709 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.334851 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.335120 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20260929 00:57:05.337777 5050 raft_consensus.cc:348] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10f932c85c64f668d72473bdfa94f10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41743 } }
I20260929 00:57:05.338021 5050 raft_consensus.cc:374] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:05.338117 5050 raft_consensus.cc:729] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10f932c85c64f668d72473bdfa94f10, State: Initialized, Role: FOLLOWER
I20260929 00:57:05.338567 5050 consensus_queue.cc:260] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10f932c85c64f668d72473bdfa94f10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41743 } }
I20260929 00:57:05.338719 5050 raft_consensus.cc:388] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:05.338805 5050 raft_consensus.cc:482] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:05.338927 5050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:05.339844 5050 raft_consensus.cc:504] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10f932c85c64f668d72473bdfa94f10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41743 } }
I20260929 00:57:05.340197 5050 leader_election.cc:302] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d10f932c85c64f668d72473bdfa94f10; no voters:
I20260929 00:57:05.340507 5050 leader_election.cc:288] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:05.340632 5055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:05.340881 5055 raft_consensus.cc:686] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [term 1 LEADER]: Becoming Leader. State: Replica: d10f932c85c64f668d72473bdfa94f10, State: Running, Role: LEADER
I20260929 00:57:05.341200 5055 consensus_queue.cc:237] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10f932c85c64f668d72473bdfa94f10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41743 } }
I20260929 00:57:05.341786 5050 sys_catalog.cc:564] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:05.343019 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d10f932c85c64f668d72473bdfa94f10. Latest consensus state: current_term: 1 leader_uuid: "d10f932c85c64f668d72473bdfa94f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10f932c85c64f668d72473bdfa94f10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41743 } } }
I20260929 00:57:05.343160 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:05.343005 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d10f932c85c64f668d72473bdfa94f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10f932c85c64f668d72473bdfa94f10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41743 } } }
I20260929 00:57:05.343375 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10 [sys.catalog]: This master's current role is: LEADER
W20260929 00:57:05.346905 5070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:57:05.347004 5070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:57:05.347103 5064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:05.347922 5064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:05.353351 5064 catalog_manager.cc:1347] Generated new cluster ID: 2fce3e6153bd41d0b5a67a6ae7d84601
I20260929 00:57:05.353435 5064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:05.368274 5064 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:05.368681 5064 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:05.374485 5064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10: Generated new TSK 0
I20260929 00:57:05.375311 5064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:05.380018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41743
--builtin_ntp_servers=127.0.98.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 00:57:05.498068 5074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:05.498447 5074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:05.498529 5074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:05.502707 5074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:05.502794 5074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:05.502887 5074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:05.508121 5074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:05.509341 5074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:05.511081 5074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:57:05.521430 5074 server_base.cc:1034] running on GCE node
W20260929 00:57:05.521359 5080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:05.521384 5082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:05.521349 5079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:05.522109 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:05.522815 5074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:05.523972 5074 hybrid_clock.cc:648] HybridClock initialized: now 1790643425523955 us; error 32 us; skew 500 ppm
I20260929 00:57:05.525797 5074 webserver.cc:460] Webserver started at http://127.0.98.1:46271/ using document root <none> and password file <none>
I20260929 00:57:05.526439 5074 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:05.526505 5074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:05.526714 5074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:05.528884 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "0169392540664741977ca15f8a94cffe"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "094814e15eedf9f7e5d938a8ee4e7086"
server_key_iv: "7d2c3486b68196e5b9f6701e464a62e3"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.529448 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0169392540664741977ca15f8a94cffe"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "094814e15eedf9f7e5d938a8ee4e7086"
server_key_iv: "7d2c3486b68196e5b9f6701e464a62e3"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.533241 5074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:57:05.535768 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.536998 5074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:57:05.537142 5074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "0169392540664741977ca15f8a94cffe"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "094814e15eedf9f7e5d938a8ee4e7086"
server_key_iv: "7d2c3486b68196e5b9f6701e464a62e3"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.537256 5074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:05.542018 5074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:05.542759 5074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:05.542918 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:05.543476 5074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:05.544456 5074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:05.544502 5074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.544539 5074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:05.544560 5074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.553292 5074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44693
I20260929 00:57:05.553314 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44693 every 8 connection(s)
I20260929 00:57:05.554214 5074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:05.558703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 5074
I20260929 00:57:05.558832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:05.559116 392 external_mini_cluster.cc:1428] Setting key 23623ecb74c7d3ddcff31282c4645aac
I20260929 00:57:05.560467 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.555968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42723 (local address 127.0.98.62:41743)
0929 00:57:05.556099 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.556103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.557184 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 00:57:05.558587 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.558590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.558648 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.558757 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:05.559984 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:05.559988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:05.559991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:05.560009 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:05.560028 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:05.560036 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:05.560155 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.560295 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.560339 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:57:05.561035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41743
--builtin_ntp_servers=127.0.98.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 00:57:05.561440 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.556379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41743 (local address 127.0.98.1:42723)
0929 00:57:05.557031 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.557071 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:05.558373 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:05.558955 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:05.558966 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:05.559071 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:05.559728 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:05.559729 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:05.559771 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:05.559839 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:05.560140 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:05.560149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:05.560155 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:57:05.560341 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 00:57:05.560605 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260929 00:57:05.562739 5202 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41743
I20260929 00:57:05.562989 5202 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:05.563472 5202 heartbeater.cc:511] Master 127.0.98.62:41743 requested a full tablet report, sending...
I20260929 00:57:05.565037 5014 ts_manager.cc:194] Registered new tserver with Master: 0169392540664741977ca15f8a94cffe (127.0.98.1:44693)
W20260929 00:57:05.672422 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:05.672767 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:05.672847 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:05.677075 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:05.677191 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:05.677289 5205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:05.682202 5205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:05.683387 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:05.684875 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:05.694479 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:05.694522 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:05.694509 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:05.694777 5205 server_base.cc:1034] running on GCE node
I20260929 00:57:05.695191 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:05.696100 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:05.697276 5205 hybrid_clock.cc:648] HybridClock initialized: now 1790643425697253 us; error 52 us; skew 500 ppm
I20260929 00:57:05.699350 5205 webserver.cc:460] Webserver started at http://127.0.98.2:43417/ using document root <none> and password file <none>
I20260929 00:57:05.700007 5205 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:05.700102 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:05.700346 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:05.702665 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "d63c61b2ef494cc9a8a5de5c308288f5"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "c29c0db910052b74c4a30cb90287f3a4"
server_key_iv: "dc226505254756e4205cb7eba2438ec2"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.703275 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d63c61b2ef494cc9a8a5de5c308288f5"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "c29c0db910052b74c4a30cb90287f3a4"
server_key_iv: "dc226505254756e4205cb7eba2438ec2"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.707352 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:57:05.710032 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.711347 5205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:05.711480 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "d63c61b2ef494cc9a8a5de5c308288f5"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "c29c0db910052b74c4a30cb90287f3a4"
server_key_iv: "dc226505254756e4205cb7eba2438ec2"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.711609 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:05.716462 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:05.717128 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:05.717322 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:05.717953 5205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:05.718988 5205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:05.719035 5205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.719117 5205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:05.719161 5205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.727932 5205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36917
I20260929 00:57:05.727950 5332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36917 every 8 connection(s)
I20260929 00:57:05.728850 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:05.735010 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.730682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59293 (local address 127.0.98.62:41743)
0929 00:57:05.730814 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.730818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.731916 (+ 1098us) server_negotiation.cc:407] Connection header received
0929 00:57:05.733210 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.733214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.733256 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.733334 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:05.734474 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:05.734478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:05.734480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:05.734493 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:05.734515 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:05.734532 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:05.734672 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.734818 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.734862 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:57:05.735837 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.731104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41743 (local address 127.0.98.2:59293)
0929 00:57:05.731742 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.731786 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:05.732986 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:05.733495 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:05.733501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:05.733587 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:05.734216 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:05.734218 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:05.734259 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:05.734326 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:05.734657 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:05.734665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:05.734671 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:57:05.734860 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 00:57:05.735105 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260929 00:57:05.736935 5333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41743
I20260929 00:57:05.737174 5333 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:05.737562 5333 heartbeater.cc:511] Master 127.0.98.62:41743 requested a full tablet report, sending...
I20260929 00:57:05.737825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 5205
I20260929 00:57:05.737953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:05.738327 392 external_mini_cluster.cc:1428] Setting key e8b627933a2f015eee89269328add98e
I20260929 00:57:05.738641 5014 ts_manager.cc:194] Registered new tserver with Master: d63c61b2ef494cc9a8a5de5c308288f5 (127.0.98.2:36917)
I20260929 00:57:05.740265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41743
--builtin_ntp_servers=127.0.98.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 00:57:05.853065 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:05.853389 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:05.853499 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:05.857581 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:05.857664 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:05.857751 5336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:05.862551 5336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:05.863634 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:05.864992 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:05.874451 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:05.874564 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:05.874482 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:05.874852 5336 server_base.cc:1034] running on GCE node
I20260929 00:57:05.875314 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:05.876035 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:05.877228 5336 hybrid_clock.cc:648] HybridClock initialized: now 1790643425877210 us; error 32 us; skew 500 ppm
I20260929 00:57:05.879135 5336 webserver.cc:460] Webserver started at http://127.0.98.3:35669/ using document root <none> and password file <none>
I20260929 00:57:05.879716 5336 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:05.879807 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:05.880051 5336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:05.882104 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "3242f4bf7fbe49dda4b64380a7e536ae"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "5e56440d02771b253189e6616877baac"
server_key_iv: "88d0245dfb5389520f9d2dbc8c07c847"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.882817 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3242f4bf7fbe49dda4b64380a7e536ae"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "5e56440d02771b253189e6616877baac"
server_key_iv: "88d0245dfb5389520f9d2dbc8c07c847"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.887084 5336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:57:05.890318 5350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.891824 5336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:57:05.891984 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "3242f4bf7fbe49dda4b64380a7e536ae"
format_stamp: "Formatted at 2026-09-29 00:57:05 on dist-test-slave-30s6"
server_key: "5e56440d02771b253189e6616877baac"
server_key_iv: "88d0245dfb5389520f9d2dbc8c07c847"
server_key_version: "encryptionkey@0"
I20260929 00:57:05.892150 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:05.896984 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:05.897617 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:05.897807 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:05.898443 5336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:05.899385 5336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:05.899458 5336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.899528 5336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:05.899581 5336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:05.908413 5336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44011
I20260929 00:57:05.908435 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44011 every 8 connection(s)
I20260929 00:57:05.909447 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:05.915699 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.911439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49741 (local address 127.0.98.62:41743)
0929 00:57:05.911570 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.911573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.912710 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 00:57:05.913928 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.913931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.913967 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.914045 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:05.915314 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:05.915317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:05.915319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:05.915331 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:05.915351 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:05.915359 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:05.915442 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.915547 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.915584 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:57:05.916468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 5336
I20260929 00:57:05.916611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:05.916855 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.911890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41743 (local address 127.0.98.3:49741)
0929 00:57:05.912542 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.912583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:05.913743 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:05.914295 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:05.914302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:05.914392 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:05.915052 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:05.915053 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:05.915100 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:05.915170 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:05.915435 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:05.915440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:05.915444 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:57:05.915569 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 00:57:05.915844 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260929 00:57:05.916976 392 external_mini_cluster.cc:1428] Setting key 747c6e27285d310f1ba3cc4b425d9086
I20260929 00:57:05.918311 5464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41743
I20260929 00:57:05.918624 5464 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:05.919220 5464 heartbeater.cc:511] Master 127.0.98.62:41743 requested a full tablet report, sending...
I20260929 00:57:05.920320 5014 ts_manager.cc:194] Registered new tserver with Master: 3242f4bf7fbe49dda4b64380a7e536ae (127.0.98.3:44011)
I20260929 00:57:05.931248 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:57:05.934700 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.933073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.0.98.62:41743)
0929 00:57:05.933219 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.933224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.933346 (+ 122us) server_negotiation.cc:407] Connection header received
0929 00:57:05.933444 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.933446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.933480 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.933571 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:05.934332 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:05.934334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:05.934336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:05.934346 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:05.934361 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:05.934368 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:05.934437 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.934497 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.934528 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:57:05.937549 5014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:05.940609 5014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:05.952476 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.950169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.0.98.2:36917)
0929 00:57:05.950488 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.950503 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.950527 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:57:05.950592 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.950597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.950775 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.951277 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:05.951919 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:05.951925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:05.951937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:05.951948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:05.951994 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:05.952025 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:05.952122 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.952234 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.952303 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
I20260929 00:57:05.952467 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.949675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36917 (local address 127.0.0.1:39260)
0929 00:57:05.950121 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.950136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:05.950241 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:05.950916 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:05.950918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:05.950994 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:05.951784 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:05.951786 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:05.951794 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:05.951808 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:05.952109 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:05.952117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:05.952122 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:57:05.952223 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:57:05.952291 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":93,"threads_started":1}
I20260929 00:57:05.952750 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.949462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.0.98.3:44011)
0929 00:57:05.949738 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.949750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.949774 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:57:05.949826 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.949833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.950040 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.950224 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:05.952117 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:05.952125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:05.952136 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:05.952151 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:05.952192 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:05.952227 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:05.952322 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.952523 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.952591 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":115,"threads_started":1}
I20260929 00:57:05.953047 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.948772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44011 (local address 127.0.0.1:42536)
0929 00:57:05.949125 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.949179 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:05.949273 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:05.950908 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:05.950914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:05.950994 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:05.951926 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:05.951928 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:05.951938 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:05.951964 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:05.952364 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:05.952367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:05.952370 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 00:57:05.952434 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 00:57:05.952462 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260929 00:57:05.953207 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.949601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.0.98.1:44693)
0929 00:57:05.950488 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:05.950500 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:05.950522 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:57:05.950575 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:05.950580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:05.950764 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:05.950913 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:05.951959 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:05.951964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:05.951972 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:05.951981 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:05.952037 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:05.952074 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:05.952543 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:05.952597 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 00:57:05.952649 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":143,"threads_started":1}
I20260929 00:57:05.952471 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:05.948962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.0.1:37276)
0929 00:57:05.949248 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:05.949262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:05.949340 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:05.950908 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:05.950914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:05.950994 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:05.951703 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:05.951705 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:05.951720 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:05.951806 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:05.952180 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:05.952185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:05.952188 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 00:57:05.952248 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 00:57:05.952293 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":125,"threads_started":1}
I20260929 00:57:05.955809 5136 tablet_service.cc:1467] Processing CreateTablet for tablet 976ef75694c940e092de94d6ffa01151 (DEFAULT_TABLE table=test-table [id=a6583d8df515480d8eaad79a08886400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:05.955946 5398 tablet_service.cc:1467] Processing CreateTablet for tablet 976ef75694c940e092de94d6ffa01151 (DEFAULT_TABLE table=test-table [id=a6583d8df515480d8eaad79a08886400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:05.955945 5267 tablet_service.cc:1467] Processing CreateTablet for tablet 976ef75694c940e092de94d6ffa01151 (DEFAULT_TABLE table=test-table [id=a6583d8df515480d8eaad79a08886400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:05.956926 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976ef75694c940e092de94d6ffa01151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:05.956926 5267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976ef75694c940e092de94d6ffa01151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:05.956935 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976ef75694c940e092de94d6ffa01151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:05.962926 5481 tablet_bootstrap.cc:492] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe: Bootstrap starting.
I20260929 00:57:05.964464 5483 tablet_bootstrap.cc:492] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Bootstrap starting.
I20260929 00:57:05.965241 5481 tablet_bootstrap.cc:654] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:05.966027 5481 log.cc:824] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:05.966483 5483 tablet_bootstrap.cc:654] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:05.966506 5482 tablet_bootstrap.cc:492] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: Bootstrap starting.
I20260929 00:57:05.967212 5483 log.cc:824] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:05.968269 5481 tablet_bootstrap.cc:492] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe: No bootstrap required, opened a new log
I20260929 00:57:05.968465 5481 ts_tablet_manager.cc:1397] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:57:05.968547 5482 tablet_bootstrap.cc:654] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:05.968920 5483 tablet_bootstrap.cc:492] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: No bootstrap required, opened a new log
I20260929 00:57:05.969110 5483 ts_tablet_manager.cc:1397] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 00:57:05.969309 5482 log.cc:824] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:05.971480 5482 tablet_bootstrap.cc:492] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: No bootstrap required, opened a new log
I20260929 00:57:05.971675 5482 ts_tablet_manager.cc:1397] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 00:57:05.972945 5483 raft_consensus.cc:348] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:05.972957 5481 raft_consensus.cc:348] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:05.973181 5483 raft_consensus.cc:374] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:05.973204 5481 raft_consensus.cc:374] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:05.973232 5483 raft_consensus.cc:729] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63c61b2ef494cc9a8a5de5c308288f5, State: Initialized, Role: FOLLOWER
I20260929 00:57:05.973261 5481 raft_consensus.cc:729] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0169392540664741977ca15f8a94cffe, State: Initialized, Role: FOLLOWER
I20260929 00:57:05.973644 5483 consensus_queue.cc:260] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:05.973716 5481 consensus_queue.cc:260] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:05.974289 5333 heartbeater.cc:503] Master 127.0.98.62:41743 was elected leader, sending a full tablet report...
I20260929 00:57:05.974660 5483 ts_tablet_manager.cc:1428] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 00:57:05.974751 5481 ts_tablet_manager.cc:1428] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 00:57:05.975085 5202 heartbeater.cc:503] Master 127.0.98.62:41743 was elected leader, sending a full tablet report...
I20260929 00:57:05.975780 5482 raft_consensus.cc:348] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:05.976034 5482 raft_consensus.cc:374] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:05.976089 5482 raft_consensus.cc:729] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3242f4bf7fbe49dda4b64380a7e536ae, State: Initialized, Role: FOLLOWER
I20260929 00:57:05.976514 5482 consensus_queue.cc:260] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:05.977190 5464 heartbeater.cc:503] Master 127.0.98.62:41743 was elected leader, sending a full tablet report...
I20260929 00:57:05.977447 5482 ts_tablet_manager.cc:1428] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 00:57:05.979827 5334 tablet.cc:2369] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:06.013940 5488 raft_consensus.cc:482] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:06.014147 5488 raft_consensus.cc:504] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:06.015375 5488 leader_election.cc:288] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3242f4bf7fbe49dda4b64380a7e536ae (127.0.98.3:44011), d63c61b2ef494cc9a8a5de5c308288f5 (127.0.98.2:36917)
I20260929 00:57:06.017266 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.015732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54969 (local address 127.0.98.3:44011)
0929 00:57:06.015868 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:06.015874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:06.015889 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:06.015936 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:06.015941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:06.015980 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:06.016065 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:06.016857 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:06.016868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:06.016870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:06.016880 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:06.016897 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:06.016906 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:06.016996 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:06.017102 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 00:57:06.017140 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:57:06.017264 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.015588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44011 (local address 127.0.98.1:54969)
0929 00:57:06.015722 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:06.015738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:06.015821 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:06.016110 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:06.016113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:06.016126 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:06.016685 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:06.016689 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:06.016704 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:06.016734 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:06.016990 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:06.016993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:06.016996 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 00:57:06.017101 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:57:06.017140 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 00:57:06.017745 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.015895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49295 (local address 127.0.98.2:36917)
0929 00:57:06.016037 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:06.016043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:06.016229 (+ 186us) server_negotiation.cc:407] Connection header received
0929 00:57:06.016344 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:06.016349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:06.016388 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:06.016469 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:06.017264 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:06.017267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:06.017270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:06.017281 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:06.017306 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:06.017334 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:06.017441 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:06.017586 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 00:57:06.017629 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:57:06.017750 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.015679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36917 (local address 127.0.98.1:49295)
0929 00:57:06.016130 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:06.016143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:06.016229 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:06.016497 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:06.016499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:06.016511 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:06.017106 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:06.017108 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:06.017117 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:06.017141 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:06.017482 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:06.017485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:06.017488 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 00:57:06.017560 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 00:57:06.017593 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":127,"threads_started":1}
I20260929 00:57:06.017961 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976ef75694c940e092de94d6ffa01151" candidate_uuid: "0169392540664741977ca15f8a94cffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" is_pre_election: true
I20260929 00:57:06.018312 5418 raft_consensus.cc:2454] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0169392540664741977ca15f8a94cffe in term 0.
I20260929 00:57:06.018440 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976ef75694c940e092de94d6ffa01151" candidate_uuid: "0169392540664741977ca15f8a94cffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" is_pre_election: true
I20260929 00:57:06.018713 5287 raft_consensus.cc:2454] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0169392540664741977ca15f8a94cffe in term 0.
I20260929 00:57:06.018774 5090 leader_election.cc:302] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0169392540664741977ca15f8a94cffe, 3242f4bf7fbe49dda4b64380a7e536ae; no voters:
I20260929 00:57:06.019088 5488 raft_consensus.cc:2781] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:06.019151 5488 raft_consensus.cc:482] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:06.019181 5488 raft_consensus.cc:3037] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:06.020196 5488 raft_consensus.cc:504] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:06.020529 5488 leader_election.cc:288] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [CANDIDATE]: Term 1 election: Requested vote from peers 3242f4bf7fbe49dda4b64380a7e536ae (127.0.98.3:44011), d63c61b2ef494cc9a8a5de5c308288f5 (127.0.98.2:36917)
I20260929 00:57:06.020869 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976ef75694c940e092de94d6ffa01151" candidate_uuid: "0169392540664741977ca15f8a94cffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63c61b2ef494cc9a8a5de5c308288f5"
I20260929 00:57:06.020907 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976ef75694c940e092de94d6ffa01151" candidate_uuid: "0169392540664741977ca15f8a94cffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3242f4bf7fbe49dda4b64380a7e536ae"
I20260929 00:57:06.020995 5287 raft_consensus.cc:3037] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:06.021023 5418 raft_consensus.cc:3037] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:06.022004 5418 raft_consensus.cc:2454] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0169392540664741977ca15f8a94cffe in term 1.
I20260929 00:57:06.022008 5287 raft_consensus.cc:2454] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0169392540664741977ca15f8a94cffe in term 1.
I20260929 00:57:06.022292 5090 leader_election.cc:302] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0169392540664741977ca15f8a94cffe, 3242f4bf7fbe49dda4b64380a7e536ae; no voters:
I20260929 00:57:06.022495 5488 raft_consensus.cc:2781] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:06.022687 5488 raft_consensus.cc:686] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 1 LEADER]: Becoming Leader. State: Replica: 0169392540664741977ca15f8a94cffe, State: Running, Role: LEADER
I20260929 00:57:06.022995 5488 consensus_queue.cc:237] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } }
I20260929 00:57:06.026175 5012 catalog_manager.cc:5495] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe reported cstate change: term changed from 0 to 1, leader changed from <none> to 0169392540664741977ca15f8a94cffe (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0169392540664741977ca15f8a94cffe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0169392540664741977ca15f8a94cffe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:06.051822 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.050257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.0.98.1:44693)
0929 00:57:06.050405 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:06.050410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:06.050424 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:57:06.050603 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:06.050606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:06.050668 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:06.050759 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:06.051409 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:06.051412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:06.051415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:06.051427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:06.051449 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:06.051463 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:06.051562 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:06.051626 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 00:57:06.051669 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 00:57:06.055270 5203 tablet.cc:2369] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:06.057921 5287 raft_consensus.cc:1261] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 1 FOLLOWER]: Refusing update from remote peer 0169392540664741977ca15f8a94cffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:06.057921 5418 raft_consensus.cc:1261] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 1 FOLLOWER]: Refusing update from remote peer 0169392540664741977ca15f8a94cffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:06.058785 5491 consensus_queue.cc:1035] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3242f4bf7fbe49dda4b64380a7e536ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:06.058959 5488 consensus_queue.cc:1035] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63c61b2ef494cc9a8a5de5c308288f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:06.070042 5496 mvcc.cc:205] Tried to move back new op lower bound from 7334475473125969920 to 7334475472992903168. Current Snapshot: MvccSnapshot[applied={T|T < 7334475473125969920}]
I20260929 00:57:06.077690 5014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:06.077839 5014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:06.080202 5014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d10f932c85c64f668d72473bdfa94f10: Sending DeleteTablet for 3 replicas of tablet 976ef75694c940e092de94d6ffa01151
I20260929 00:57:06.080866 5398 tablet_service.cc:1514] Processing DeleteTablet for tablet 976ef75694c940e092de94d6ffa01151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:06 UTC) from {username='slave'} at 127.0.0.1:42536
I20260929 00:57:06.080922 5136 tablet_service.cc:1514] Processing DeleteTablet for tablet 976ef75694c940e092de94d6ffa01151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:06 UTC) from {username='slave'} at 127.0.0.1:37276
I20260929 00:57:06.081023 5267 tablet_service.cc:1514] Processing DeleteTablet for tablet 976ef75694c940e092de94d6ffa01151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:06 UTC) from {username='slave'} at 127.0.0.1:39260
I20260929 00:57:06.081390 5506 tablet_replica.cc:330] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe: stopping tablet replica
I20260929 00:57:06.081715 5507 tablet_replica.cc:330] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: stopping tablet replica
I20260929 00:57:06.081720 5506 raft_consensus.cc:2229] T 976ef75694c940e092de94d6ffa01151 P 0169392540664741977ca15f8a94cffe [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:06.082011 5507 raft_consensus.cc:2229] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:06.082046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 5074
I20260929 00:57:06.082340 5507 raft_consensus.cc:2258] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:06.082894 5505 tablet_replica.cc:330] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: stopping tablet replica
I20260929 00:57:06.083181 5505 raft_consensus.cc:2229] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:06.083447 5505 raft_consensus.cc:2258] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:06.084089 5507 ts_tablet_manager.cc:1905] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:06.084921 5505 ts_tablet_manager.cc:1905] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:06.086597 5507 ts_tablet_manager.cc:1918] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:06.086704 5507 log.cc:1196] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/976ef75694c940e092de94d6ffa01151
I20260929 00:57:06.086988 5507 ts_tablet_manager.cc:1939] T 976ef75694c940e092de94d6ffa01151 P d63c61b2ef494cc9a8a5de5c308288f5: Deleting consensus metadata
I20260929 00:57:06.088153 5000 catalog_manager.cc:4841] TS d63c61b2ef494cc9a8a5de5c308288f5 (127.0.98.2:36917): tablet 976ef75694c940e092de94d6ffa01151 (table test-table [id=a6583d8df515480d8eaad79a08886400]) successfully deleted
I20260929 00:57:06.088357 5505 ts_tablet_manager.cc:1918] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:06.088461 5505 log.cc:1196] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/976ef75694c940e092de94d6ffa01151
I20260929 00:57:06.088784 5505 ts_tablet_manager.cc:1939] T 976ef75694c940e092de94d6ffa01151 P 3242f4bf7fbe49dda4b64380a7e536ae: Deleting consensus metadata
I20260929 00:57:06.090013 4999 catalog_manager.cc:4841] TS 3242f4bf7fbe49dda4b64380a7e536ae (127.0.98.3:44011): tablet 976ef75694c940e092de94d6ffa01151 (table test-table [id=a6583d8df515480d8eaad79a08886400]) successfully deleted
W20260929 00:57:06.090458 5001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44693 (error 108)
I20260929 00:57:06.091245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 5205
I20260929 00:57:06.091475 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.091164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.0.1:37300)
0929 00:57:06.091305 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:06.091382 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 00:57:06.092082 5001 catalog_manager.cc:4584] TS 0169392540664741977ca15f8a94cffe (127.0.98.1:44693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 976ef75694c940e092de94d6ffa01151: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44693: connect: Connection refused (error 111)
I20260929 00:57:06.097162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 5336
I20260929 00:57:06.102851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 4983
2026-09-29T00:57:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (991 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 00:57:06.124382 392 mini_oidc.cc:152] Starting JWKS server
I20260929 00:57:06.125972 392 webserver.cc:460] Webserver started at https://127.0.0.1:45243/ using document root <none> and password file <none>
I20260929 00:57:06.126163 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:57:06.126475 392 webserver.cc:460] Webserver started at http://127.0.0.1:38597/ using document root <none> and password file <none>
2026-09-29T00:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:06Z Disabled control of system clock
I20260929 00:57:06.135327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33057
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33057
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45243/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/testchainca.cert with env {}
W20260929 00:57:06.246352 5514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:06.246855 5514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:06.246953 5514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:06.251178 5514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:06.251238 5514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:06.251253 5514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:06.251292 5514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:06.251325 5514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:57:06.251348 5514 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:57:06.251374 5514 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45243/jwks.json
I20260929 00:57:06.256817 5514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45243/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:06.257885 5514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:06.258991 5514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:06.267366 5522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:06.267410 5514 server_base.cc:1034] running on GCE node
W20260929 00:57:06.267516 5520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:06.267334 5519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:06.269312 5514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:06.270645 5514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:06.271833 5514 hybrid_clock.cc:648] HybridClock initialized: now 1790643426271808 us; error 54 us; skew 500 ppm
I20260929 00:57:06.273890 5514 webserver.cc:460] Webserver started at http://127.0.98.62:45783/ using document root <none> and password file <none>
I20260929 00:57:06.274531 5514 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:06.274622 5514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:06.274850 5514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:06.276870 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "97f0ac65ff224308b8d8ee70f68021cc"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "b25e486fb9fd3fe192afc24f6da25ea8"
server_key_iv: "7fa99c8593a180c269db3d5e1d862068"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.277434 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "97f0ac65ff224308b8d8ee70f68021cc"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "b25e486fb9fd3fe192afc24f6da25ea8"
server_key_iv: "7fa99c8593a180c269db3d5e1d862068"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.281081 5514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 00:57:06.283555 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:06.284796 5514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:57:06.284933 5514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "97f0ac65ff224308b8d8ee70f68021cc"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "b25e486fb9fd3fe192afc24f6da25ea8"
server_key_iv: "7fa99c8593a180c269db3d5e1d862068"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.285043 5514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:06.301633 5514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:06.302399 5514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:06.302606 5514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:06.310122 5514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33057
I20260929 00:57:06.310122 5580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33057 every 8 connection(s)
I20260929 00:57:06.311192 5514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:06.314502 5581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:06.319900 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc: Bootstrap starting.
I20260929 00:57:06.320914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 5514
I20260929 00:57:06.321053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:06.321360 392 external_mini_cluster.cc:1428] Setting key 9874624593d715cbb885e86547887482
I20260929 00:57:06.322932 5581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:06.323670 5581 log.cc:824] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:06.325606 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc: No bootstrap required, opened a new log
I20260929 00:57:06.329107 5581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f0ac65ff224308b8d8ee70f68021cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33057 } }
I20260929 00:57:06.329336 5581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:06.329398 5581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f0ac65ff224308b8d8ee70f68021cc, State: Initialized, Role: FOLLOWER
I20260929 00:57:06.329932 5581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f0ac65ff224308b8d8ee70f68021cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33057 } }
I20260929 00:57:06.330081 5581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:06.330168 5581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:06.330297 5581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:06.330967 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.322434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.0.98.62:33057)
0929 00:57:06.323150 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:06.323166 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:06.323199 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:57:06.323896 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:06.323915 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:06.324368 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:06.324747 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:06.325172 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.326285 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.327277 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.327846 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.328014 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:06.328883 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:06.328889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:06.328911 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:06.329356 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:06.329449 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:06.329478 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:06.329576 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:06.329685 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 00:57:06.329957 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":147,"threads_started":1}
I20260929 00:57:06.331334 5581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f0ac65ff224308b8d8ee70f68021cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33057 } }
I20260929 00:57:06.331696 5581 leader_election.cc:302] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97f0ac65ff224308b8d8ee70f68021cc; no voters:
I20260929 00:57:06.332041 5581 leader_election.cc:288] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:06.332162 5586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:06.332345 5586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [term 1 LEADER]: Becoming Leader. State: Replica: 97f0ac65ff224308b8d8ee70f68021cc, State: Running, Role: LEADER
I20260929 00:57:06.332647 5586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f0ac65ff224308b8d8ee70f68021cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33057 } }
I20260929 00:57:06.333371 5581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:06.333922 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97f0ac65ff224308b8d8ee70f68021cc. Latest consensus state: current_term: 1 leader_uuid: "97f0ac65ff224308b8d8ee70f68021cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f0ac65ff224308b8d8ee70f68021cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33057 } } }
I20260929 00:57:06.334046 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:06.334416 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:57:06.334343 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97f0ac65ff224308b8d8ee70f68021cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f0ac65ff224308b8d8ee70f68021cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33057 } } }
I20260929 00:57:06.335388 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:06.334476 5591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:06.338812 5591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:06.344260 5591 catalog_manager.cc:1347] Generated new cluster ID: 0aec4002e0dc4d9393712dbc8a250260
I20260929 00:57:06.344343 5591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:06.352473 5591 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:06.353464 5591 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:06.358598 5591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97f0ac65ff224308b8d8ee70f68021cc: Generated new TSK 0
I20260929 00:57:06.359455 5591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 00:57:06.394938 5611 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 00:57:06.395069 5611 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 00:57:06.395568 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.371388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.0.98.62:33057)
0929 00:57:06.371557 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:06.371563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:06.371748 (+ 185us) server_negotiation.cc:407] Connection header received
0929 00:57:06.372000 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:06.372003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:06.372061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:06.372164 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:57:06.372419 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.372971 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.373876 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.374124 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.374305 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:06.374674 (+ 369us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:57:06.395099 (+ 20425us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 00:57:06.395402 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49898: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 00:57:06.395670 5584 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49898: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 00:57:06.396884 5610 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:06.371302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33057 (local address 127.0.0.1:49898)
0929 00:57:06.371661 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:06.371673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:06.371820 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:06.372191 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:06.372195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:06.372203 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:57:06.372308 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:06.372314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.373091 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.373095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:06.373750 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:06.373758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.374301 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.374327 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:06.374426 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:06.374472 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:57:06.395357 (+ 20885us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 00:57:06.395606 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33057: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33057: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 00:57:06.397745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 5514
2026-09-29T00:57:06Z chronyd exiting
I20260929 00:57:06.435276 392 test_util.cc:175] -----------------------------------------------
I20260929 00:57:06.435364 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643398448434-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 00:57:06.436396 392 mini_oidc.cc:152] Starting JWKS server
I20260929 00:57:06.437214 392 webserver.cc:460] Webserver started at https://127.0.0.1:42697/ using document root <none> and password file <none>
I20260929 00:57:06.437307 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:57:06.437563 392 webserver.cc:460] Webserver started at http://127.0.0.1:33885/ using document root <none> and password file <none>
2026-09-29T00:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:06Z Disabled control of system clock
I20260929 00:57:06.446290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36815
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36815
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/testchainca.cert with env {}
W20260929 00:57:06.557897 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:06.558204 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:06.558248 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:06.562347 5617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:06.562404 5617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:06.562420 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:06.562438 5617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:06.562453 5617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:57:06.562471 5617 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:57:06.562485 5617 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42697/jwks.json
I20260929 00:57:06.567845 5617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:06.568902 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:06.569868 5617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:06.578598 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:06.578651 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:06.578675 5617 server_base.cc:1034] running on GCE node
W20260929 00:57:06.578615 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:06.579440 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:06.580511 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:06.581676 5617 hybrid_clock.cc:648] HybridClock initialized: now 1790643426581658 us; error 32 us; skew 500 ppm
I20260929 00:57:06.583508 5617 webserver.cc:460] Webserver started at http://127.0.98.62:44967/ using document root <none> and password file <none>
I20260929 00:57:06.584043 5617 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:06.584101 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:06.584254 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:06.586301 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "a5507e0979f4422d9cbf0dec13432b77"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "f25b834575318d32f8b9399e8b34d6cc"
server_key_iv: "c465716caf9b9302aeb6df9a58b0c1dc"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.586838 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5507e0979f4422d9cbf0dec13432b77"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "f25b834575318d32f8b9399e8b34d6cc"
server_key_iv: "c465716caf9b9302aeb6df9a58b0c1dc"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.590533 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:57:06.592940 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:06.594297 5617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:06.594422 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "a5507e0979f4422d9cbf0dec13432b77"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "f25b834575318d32f8b9399e8b34d6cc"
server_key_iv: "c465716caf9b9302aeb6df9a58b0c1dc"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.594537 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:06.623956 5617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:06.624698 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:06.624917 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:06.632364 5617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36815
I20260929 00:57:06.632426 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36815 every 8 connection(s)
I20260929 00:57:06.633333 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:06.636541 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:06.642084 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77: Bootstrap starting.
I20260929 00:57:06.642380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 5617
I20260929 00:57:06.642498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:06.642779 392 external_mini_cluster.cc:1428] Setting key d871a96f5f1ba718d29313b4a11efce6
I20260929 00:57:06.644763 5684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:06.645519 5684 log.cc:824] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:06.647553 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77: No bootstrap required, opened a new log
I20260929 00:57:06.652034 5684 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5507e0979f4422d9cbf0dec13432b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } }
I20260929 00:57:06.652295 5684 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:06.652374 5684 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5507e0979f4422d9cbf0dec13432b77, State: Initialized, Role: FOLLOWER
I20260929 00:57:06.652817 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.644119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.0.98.62:36815)
0929 00:57:06.644736 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:06.644750 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:06.644785 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:57:06.645462 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:06.645479 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:06.645967 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:06.646374 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:06.646721 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.647712 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.648942 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.649438 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.649570 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:06.650628 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:06.650634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:06.650647 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:06.651138 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:06.651203 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:06.651225 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:06.651317 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:06.651528 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 00:57:06.651890 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":135,"threads_started":1}
I20260929 00:57:06.652819 5684 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5507e0979f4422d9cbf0dec13432b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } }
I20260929 00:57:06.653132 5684 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:06.653203 5684 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:06.653296 5684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:06.654384 5684 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5507e0979f4422d9cbf0dec13432b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } }
I20260929 00:57:06.654719 5684 leader_election.cc:302] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5507e0979f4422d9cbf0dec13432b77; no voters:
I20260929 00:57:06.655115 5684 leader_election.cc:288] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:06.655210 5689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:06.655444 5689 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [term 1 LEADER]: Becoming Leader. State: Replica: a5507e0979f4422d9cbf0dec13432b77, State: Running, Role: LEADER
I20260929 00:57:06.655764 5689 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5507e0979f4422d9cbf0dec13432b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } }
I20260929 00:57:06.655977 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:57:06.656437 5684 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:06.656808 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5507e0979f4422d9cbf0dec13432b77. Latest consensus state: current_term: 1 leader_uuid: "a5507e0979f4422d9cbf0dec13432b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5507e0979f4422d9cbf0dec13432b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } } }
I20260929 00:57:06.656916 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:06.657243 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5507e0979f4422d9cbf0dec13432b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5507e0979f4422d9cbf0dec13432b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } } }
I20260929 00:57:06.657337 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:06.658244 5693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:06.661259 5693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:06.666811 5693 catalog_manager.cc:1347] Generated new cluster ID: 128851f2df374cde93130df435d615da
I20260929 00:57:06.666900 5693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:06.677349 5693 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:06.678460 5693 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:06.683990 5693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5507e0979f4422d9cbf0dec13432b77: Generated new TSK 0
I20260929 00:57:06.684679 5693 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 00:57:06.713479 5714 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 00:57:06.713596 5714 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 00:57:06.714332 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.697676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.0.98.62:36815)
0929 00:57:06.697826 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:06.697832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:06.698037 (+ 205us) server_negotiation.cc:407] Connection header received
0929 00:57:06.698203 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:06.698206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:06.698268 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:06.698388 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:57:06.698646 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.699524 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.700457 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.700801 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.700900 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:06.701206 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:57:06.713842 (+ 12636us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 00:57:06.714156 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 00:57:06.714401 5687 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 00:57:06.714448 5713 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:06.697601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36815 (local address 127.0.0.1:48894)
0929 00:57:06.697940 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:06.697954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:06.698052 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:06.698409 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:06.698411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:06.698419 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:57:06.698524 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:06.698529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.699658 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.699661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:06.700328 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:06.700337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.700910 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.700933 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:06.700996 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:06.701020 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:57:06.714061 (+ 13041us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 00:57:06.714268 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":124,"threads_started":1}
I20260929 00:57:06.715340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 5617
2026-09-29T00:57:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 00:57:06.754595 392 mini_oidc.cc:152] Starting JWKS server
I20260929 00:57:06.755494 392 webserver.cc:460] Webserver started at https://127.0.0.1:41299/ using document root <none> and password file <none>
I20260929 00:57:06.755631 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:57:06.755904 392 webserver.cc:460] Webserver started at http://127.0.0.1:38363/ using document root <none> and password file <none>
2026-09-29T00:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:06Z Disabled control of system clock
I20260929 00:57:06.765033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44105
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44105
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41299/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 00:57:06.879719 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:06.880110 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:06.880157 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:06.884212 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:06.884271 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:06.884287 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:06.884305 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:06.884321 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:57:06.884383 5720 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:57:06.884402 5720 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41299/jwks.json
I20260929 00:57:06.889212 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41299/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:06.890455 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:06.891883 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:06.900571 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:06.900571 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:06.900589 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:06.900722 5720 server_base.cc:1034] running on GCE node
I20260929 00:57:06.901499 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:06.902789 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:06.903990 5720 hybrid_clock.cc:648] HybridClock initialized: now 1790643426903966 us; error 51 us; skew 500 ppm
I20260929 00:57:06.905956 5720 webserver.cc:460] Webserver started at http://127.0.98.62:43149/ using document root <none> and password file <none>
I20260929 00:57:06.906618 5720 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:06.906713 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:06.906953 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:06.909274 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "ee9f3fbb9bb74db6b844c0a96c7af590"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "cdd31e1cc7df02e4f319ec2abeed4e04"
server_key_iv: "21c121a77d0a759c4019c5e219c5727d"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.909909 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee9f3fbb9bb74db6b844c0a96c7af590"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "cdd31e1cc7df02e4f319ec2abeed4e04"
server_key_iv: "21c121a77d0a759c4019c5e219c5727d"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.913991 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:57:06.916852 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:06.918157 5720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:06.918362 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "ee9f3fbb9bb74db6b844c0a96c7af590"
format_stamp: "Formatted at 2026-09-29 00:57:06 on dist-test-slave-30s6"
server_key: "cdd31e1cc7df02e4f319ec2abeed4e04"
server_key_iv: "21c121a77d0a759c4019c5e219c5727d"
server_key_version: "encryptionkey@0"
I20260929 00:57:06.918502 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:06.933825 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:06.934657 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:06.934894 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:06.942147 5720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44105
I20260929 00:57:06.942154 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44105 every 8 connection(s)
I20260929 00:57:06.943197 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:06.946437 5787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:06.951301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 5720
I20260929 00:57:06.951439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:06.951735 392 external_mini_cluster.cc:1428] Setting key e7f93436edf528ced933c60094c7642e
I20260929 00:57:06.952240 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590: Bootstrap starting.
I20260929 00:57:06.955057 5787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:06.955945 5787 log.cc:824] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:06.958057 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590: No bootstrap required, opened a new log
I20260929 00:57:06.962252 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:06.953120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.98.62:44105)
0929 00:57:06.953832 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:06.953851 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:06.953894 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:57:06.954742 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:06.954765 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:06.955304 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:06.955668 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:06.956162 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.957499 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.958461 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:06.959119 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:06.959268 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:06.960107 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:06.960113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:06.960130 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:06.960629 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:06.960696 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:06.960721 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:06.960818 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:06.960913 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 00:57:06.961187 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260929 00:57:06.962860 5787 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9f3fbb9bb74db6b844c0a96c7af590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } }
I20260929 00:57:06.963114 5787 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:06.963174 5787 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9f3fbb9bb74db6b844c0a96c7af590, State: Initialized, Role: FOLLOWER
I20260929 00:57:06.963645 5787 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9f3fbb9bb74db6b844c0a96c7af590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } }
I20260929 00:57:06.963795 5787 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:06.963855 5787 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:06.963946 5787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:06.965123 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:57:06.965103 5787 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9f3fbb9bb74db6b844c0a96c7af590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } }
I20260929 00:57:06.965451 5787 leader_election.cc:302] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee9f3fbb9bb74db6b844c0a96c7af590; no voters:
I20260929 00:57:06.965796 5787 leader_election.cc:288] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:06.965925 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:06.966172 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [term 1 LEADER]: Becoming Leader. State: Replica: ee9f3fbb9bb74db6b844c0a96c7af590, State: Running, Role: LEADER
I20260929 00:57:06.966517 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9f3fbb9bb74db6b844c0a96c7af590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } }
I20260929 00:57:06.967100 5787 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:06.967727 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9f3fbb9bb74db6b844c0a96c7af590. Latest consensus state: current_term: 1 leader_uuid: "ee9f3fbb9bb74db6b844c0a96c7af590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9f3fbb9bb74db6b844c0a96c7af590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } } }
I20260929 00:57:06.967849 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:06.968258 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee9f3fbb9bb74db6b844c0a96c7af590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9f3fbb9bb74db6b844c0a96c7af590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } } }
I20260929 00:57:06.968356 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:06.969712 5799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:06.972316 5799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:06.977694 5799 catalog_manager.cc:1347] Generated new cluster ID: fff9c53915c943478d292247498d6f6a
I20260929 00:57:06.977805 5799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:06.994927 5799 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:06.995916 5799 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:07.002400 5799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee9f3fbb9bb74db6b844c0a96c7af590: Generated new TSK 0
I20260929 00:57:07.003155 5799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:07.018899 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:07.014055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.0.98.62:44105)
0929 00:57:07.014205 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:07.014211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:07.014385 (+ 174us) server_negotiation.cc:407] Connection header received
0929 00:57:07.014600 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:07.014602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:07.014650 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:07.014760 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:57:07.015025 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.015571 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.016456 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.016687 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.016796 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:07.017115 (+ 319us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:57:07.018541 (+ 1426us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41299/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 00:57:07.018763 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41299/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 00:57:07.018971 5790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41299/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 00:57:07.020794 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:07.013964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44105 (local address 127.0.0.1:34976)
0929 00:57:07.014294 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:07.014306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:07.014433 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:07.014770 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:07.014773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:07.014782 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:57:07.014909 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:07.014915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.015687 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.015691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:07.016331 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:07.016338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.016788 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.016811 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:07.016896 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:07.016928 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:57:07.018715 (+ 1787us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41299/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 00:57:07.020614 (+ 1899us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44105: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41299/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1640,"thread_start_us":109,"threads_started":1}
I20260929 00:57:07.021150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 5720
E20260929 00:57:07.030009 5817 webserver.cc:533] Webserver: sslize failed:
E20260929 00:57:07.030089 5817 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T00:57:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T00:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:07Z Disabled control of system clock
I20260929 00:57:07.068589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33351
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33351
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/testchainca.cert with env {}
W20260929 00:57:07.186276 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:07.186578 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:07.186625 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:07.190923 5821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:07.191004 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:07.191020 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:07.191036 5821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:07.191103 5821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:57:07.191130 5821 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:57:07.191148 5821 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 00:57:07.196899 5821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:07.198076 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:07.199194 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:07.208884 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:07.208811 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:07.208811 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:07.209156 5821 server_base.cc:1034] running on GCE node
I20260929 00:57:07.210094 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:07.211419 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:07.212661 5821 hybrid_clock.cc:648] HybridClock initialized: now 1790643427212627 us; error 75 us; skew 500 ppm
I20260929 00:57:07.214991 5821 webserver.cc:460] Webserver started at http://127.0.98.62:33823/ using document root <none> and password file <none>
I20260929 00:57:07.215641 5821 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:07.215742 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:07.215977 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:07.218150 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "afff6c517ff54ca799d363748b598f3a"
format_stamp: "Formatted at 2026-09-29 00:57:07 on dist-test-slave-30s6"
server_key: "379e12635d9b482c2d41360a1e69b76d"
server_key_iv: "8530e774125797272caddd569be3b3c3"
server_key_version: "encryptionkey@0"
I20260929 00:57:07.218847 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "afff6c517ff54ca799d363748b598f3a"
format_stamp: "Formatted at 2026-09-29 00:57:07 on dist-test-slave-30s6"
server_key: "379e12635d9b482c2d41360a1e69b76d"
server_key_iv: "8530e774125797272caddd569be3b3c3"
server_key_version: "encryptionkey@0"
I20260929 00:57:07.222816 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:57:07.225519 5835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:07.226977 5821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:07.227154 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "afff6c517ff54ca799d363748b598f3a"
format_stamp: "Formatted at 2026-09-29 00:57:07 on dist-test-slave-30s6"
server_key: "379e12635d9b482c2d41360a1e69b76d"
server_key_iv: "8530e774125797272caddd569be3b3c3"
server_key_version: "encryptionkey@0"
I20260929 00:57:07.227293 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:07.245399 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:07.246163 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:07.246436 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:07.254401 5821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33351
I20260929 00:57:07.254428 5887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33351 every 8 connection(s)
I20260929 00:57:07.255399 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:07.258579 5888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:07.264060 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a: Bootstrap starting.
I20260929 00:57:07.265101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 5821
I20260929 00:57:07.265240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:07.265537 392 external_mini_cluster.cc:1428] Setting key 1db4384977b16206076b1c2034439d47
I20260929 00:57:07.267171 5888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:07.267970 5888 log.cc:824] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:07.270277 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a: No bootstrap required, opened a new log
I20260929 00:57:07.274695 5888 raft_consensus.cc:348] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff6c517ff54ca799d363748b598f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } }
I20260929 00:57:07.274919 5888 raft_consensus.cc:374] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:07.274977 5888 raft_consensus.cc:729] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afff6c517ff54ca799d363748b598f3a, State: Initialized, Role: FOLLOWER
I20260929 00:57:07.275220 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:07.266714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.0.98.62:33351)
0929 00:57:07.267425 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:07.267440 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:07.267485 (+ 45us) server_negotiation.cc:407] Connection header received
0929 00:57:07.268204 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:07.268223 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:07.268706 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:07.269053 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:07.269403 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.270492 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.271676 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.272157 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.272277 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:07.273243 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:07.273248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:07.273261 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:07.273670 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:07.273733 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:07.273754 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:07.273829 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:07.274021 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 00:57:07.274345 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":158,"threads_started":1}
I20260929 00:57:07.275458 5888 consensus_queue.cc:260] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff6c517ff54ca799d363748b598f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } }
I20260929 00:57:07.275609 5888 raft_consensus.cc:388] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:07.275695 5888 raft_consensus.cc:482] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:07.275813 5888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:07.276719 5888 raft_consensus.cc:504] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff6c517ff54ca799d363748b598f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } }
I20260929 00:57:07.277100 5888 leader_election.cc:302] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afff6c517ff54ca799d363748b598f3a; no voters:
I20260929 00:57:07.277439 5888 leader_election.cc:288] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:07.277570 5893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:07.277779 5893 raft_consensus.cc:686] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [term 1 LEADER]: Becoming Leader. State: Replica: afff6c517ff54ca799d363748b598f3a, State: Running, Role: LEADER
I20260929 00:57:07.278075 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:57:07.278084 5893 consensus_queue.cc:237] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff6c517ff54ca799d363748b598f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } }
I20260929 00:57:07.278702 5888 sys_catalog.cc:564] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:07.279757 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader afff6c517ff54ca799d363748b598f3a. Latest consensus state: current_term: 1 leader_uuid: "afff6c517ff54ca799d363748b598f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff6c517ff54ca799d363748b598f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } } }
I20260929 00:57:07.279893 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:07.280254 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afff6c517ff54ca799d363748b598f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff6c517ff54ca799d363748b598f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } } }
I20260929 00:57:07.280349 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:07.280642 5902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:07.283344 5902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:07.288594 5902 catalog_manager.cc:1347] Generated new cluster ID: be0dc6a8809f41dd90bdc57f806e464a
I20260929 00:57:07.288667 5902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:07.314697 5902 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:07.315663 5902 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:07.322276 5902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afff6c517ff54ca799d363748b598f3a: Generated new TSK 0
I20260929 00:57:07.323014 5902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:07.771972 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:07.332177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.0.98.62:33351)
0929 00:57:07.332310 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:07.332313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:07.332686 (+ 373us) server_negotiation.cc:407] Connection header received
0929 00:57:07.332818 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:07.332821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:07.332879 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:07.332994 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:57:07.333212 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.334039 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.334959 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.335310 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.335419 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:07.335765 (+ 346us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:57:07.771576 (+435811us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 00:57:07.771838 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 00:57:07.772047 5891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 00:57:07.772241 5917 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:07.332250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33351 (local address 127.0.0.1:35622)
0929 00:57:07.332575 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:07.332589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:07.332702 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:07.332999 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:07.333001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:07.333007 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:57:07.333102 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:07.333107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.334162 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.334165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:07.334840 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:07.334849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.335433 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.335456 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:07.335540 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:07.335586 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:57:07.771802 (+436216us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 00:57:07.771998 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33351: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":149,"threads_started":1}
I20260929 00:57:07.772828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 5821
2026-09-29T00:57:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (732 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:08 dist-test-slave-30s6 krb5kdc[5929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:08 dist-test-slave-30s6 krb5kdc[5929](info): set up 2 sockets
Sep 29 00:57:08 dist-test-slave-30s6 krb5kdc[5929](info): commencing operation
krb5kdc: starting...
W20260929 00:57:09.082509 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:09 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:57:09.219026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33551
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:09.334916 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:09.335222 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:09.335269 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:09.339484 5942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:09.339560 5942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:09.339579 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:09.339597 5942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:09.339613 5942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:09.345222 5942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:09.346912 5942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:09.348105 5942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:57:09.357432 5942 server_base.cc:1034] running on GCE node
W20260929 00:57:09.357357 5947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:09.357364 5948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:09.357437 5950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:09.358429 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:09.359719 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:09.360962 5942 hybrid_clock.cc:648] HybridClock initialized: now 1790643429360939 us; error 36 us; skew 500 ppm
Sep 29 00:57:09 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:09.364051 5942 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:57:09.364960 5942 webserver.cc:460] Webserver started at http://127.0.98.62:41509/ using document root <none> and password file <none>
I20260929 00:57:09.365495 5942 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:09.365554 5942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:09.365736 5942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:09.367904 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "4ef054b715394752a9d1de3550f53476"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "931b4fb700dac1a1e7a1e6e65b170004"
server_key_iv: "c1b6bc53d2d441729f2d40f3a361a941"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.368448 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ef054b715394752a9d1de3550f53476"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "931b4fb700dac1a1e7a1e6e65b170004"
server_key_iv: "c1b6bc53d2d441729f2d40f3a361a941"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.372450 5942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:09.375110 5957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.376571 5942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:09.376693 5942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "4ef054b715394752a9d1de3550f53476"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "931b4fb700dac1a1e7a1e6e65b170004"
server_key_iv: "c1b6bc53d2d441729f2d40f3a361a941"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.376798 5942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:09.389963 5942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:09.393028 5942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:09.393190 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:09.400715 5942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33551
I20260929 00:57:09.401080 6009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33551 every 8 connection(s)
I20260929 00:57:09.401751 5942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:09.404609 6010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:09.406334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 5942
I20260929 00:57:09.406486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:09.406824 392 external_mini_cluster.cc:1428] Setting key b931659d2af0eb8bcd8bcccc713d2a2e
I20260929 00:57:09.411384 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476: Bootstrap starting.
I20260929 00:57:09.414055 6010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:09.415035 6010 log.cc:824] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:57:09 dist-test-slave-30s6 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:09.417304 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476: No bootstrap required, opened a new log
I20260929 00:57:09.421418 6010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef054b715394752a9d1de3550f53476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260929 00:57:09.421656 6010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:09.421690 6010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef054b715394752a9d1de3550f53476, State: Initialized, Role: FOLLOWER
I20260929 00:57:09.421906 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:09.408334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.0.98.62:33551)
0929 00:57:09.408749 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:09.408765 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:09.408800 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:57:09.409449 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:09.409466 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:09.409802 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:09.410095 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:09.411070 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.412119 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.413288 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.413765 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.413885 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:09.416848 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:09.416853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:09.416867 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:09.416893 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:09.418927 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:09.419394 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:09.419407 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:09.419413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:09.419524 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:09.419813 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:09.419816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:09.419818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:09.420245 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:09.420475 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:09.420735 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 00:57:09.421032 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":139,"threads_started":1}
I20260929 00:57:09.422254 6010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef054b715394752a9d1de3550f53476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260929 00:57:09.422425 6010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:09.422485 6010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:09.422578 6010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:09.423605 6010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef054b715394752a9d1de3550f53476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260929 00:57:09.423959 6010 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef054b715394752a9d1de3550f53476; no voters:
I20260929 00:57:09.424319 6010 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:09.424381 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:09.424656 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef054b715394752a9d1de3550f53476, State: Running, Role: LEADER
I20260929 00:57:09.424993 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef054b715394752a9d1de3550f53476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260929 00:57:09.425617 6010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:09.426651 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ef054b715394752a9d1de3550f53476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef054b715394752a9d1de3550f53476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } } }
I20260929 00:57:09.426783 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:09.427100 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef054b715394752a9d1de3550f53476. Latest consensus state: current_term: 1 leader_uuid: "4ef054b715394752a9d1de3550f53476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef054b715394752a9d1de3550f53476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } } }
I20260929 00:57:09.427179 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:09.427258 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:09.430112 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:09.436277 6024 catalog_manager.cc:1347] Generated new cluster ID: 52a63b3ba1884d679b37c57d6d387cd1
I20260929 00:57:09.436362 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:09.445521 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:09.446735 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:09.456625 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ef054b715394752a9d1de3550f53476: Generated new TSK 0
I20260929 00:57:09.457510 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:57:09.498777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:09.613565 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:09.613857 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:09.613901 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:09.617977 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:09.618045 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:09.618106 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:09.624370 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:09.625505 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:09.626811 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:09.637552 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:09.637562 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:09.637542 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:09.638047 6036 server_base.cc:1034] running on GCE node
I20260929 00:57:09.638515 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:09.639243 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:09.640458 6036 hybrid_clock.cc:648] HybridClock initialized: now 1790643429640438 us; error 35 us; skew 500 ppm
Sep 29 00:57:09 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:09.643618 6036 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:57:09.644573 6036 webserver.cc:460] Webserver started at http://127.0.98.1:43083/ using document root <none> and password file <none>
I20260929 00:57:09.645176 6036 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:09.645243 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:09.645495 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:09.647801 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "5529f743cceb4762a4ee778b8b51f0c9"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "fe6945c1b958a780ef1885ee03f7bbb0"
server_key_iv: "cf3a0918c81e9866a1fd23b1d5245a55"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.648428 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5529f743cceb4762a4ee778b8b51f0c9"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "fe6945c1b958a780ef1885ee03f7bbb0"
server_key_iv: "cf3a0918c81e9866a1fd23b1d5245a55"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.652657 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:09.655893 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.657320 6036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:57:09.657492 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "5529f743cceb4762a4ee778b8b51f0c9"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "fe6945c1b958a780ef1885ee03f7bbb0"
server_key_iv: "cf3a0918c81e9866a1fd23b1d5245a55"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.657629 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:09.678610 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:09.682547 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:09.682766 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:09.683395 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:09.684249 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:09.684321 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.684412 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:09.684463 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.693291 6036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35091
I20260929 00:57:09.693310 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35091 every 8 connection(s)
I20260929 00:57:09.694288 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:09.696607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 6036
I20260929 00:57:09.696769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:09.697072 392 external_mini_cluster.cc:1428] Setting key d4436feb93728daac532afc429dd919a
Sep 29 00:57:09 dist-test-slave-30s6 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 00:57:09.711161 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:09.696291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36845 (local address 127.0.98.62:33551)
0929 00:57:09.696518 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:09.696523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:09.697545 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 00:57:09.699059 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:09.699063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:09.699112 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:09.699190 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:09.700762 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.701316 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.703266 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.703539 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.703638 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:09.706815 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:09.706818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:09.706821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:09.706849 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:09.708523 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:09.709194 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:09.709198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:09.709202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:09.709294 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:09.709748 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:09.709751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:09.709753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:09.709957 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:09.710099 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:09.710909 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 00:57:09.711033 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 00:57:09.712394 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:09.696810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.1:36845)
0929 00:57:09.697360 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:09.697415 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:09.698829 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:09.699377 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:09.699385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:09.699756 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:09.700539 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:09.700561 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.701477 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.701484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:09.703093 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:09.703108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.703730 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.703737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:09.704000 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:09.704715 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:09.704739 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:09.706592 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:09.708665 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:09.708672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:09.708691 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:09.709034 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:09.709401 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:09.709404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:09.709406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:09.709578 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:09.710147 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:09.710153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:09.710579 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 00:57:09.710812 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 00:57:09.711222 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":130,"threads_started":1}
I20260929 00:57:09.713917 6165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260929 00:57:09.714251 6165 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:09.714948 6165 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260929 00:57:09.716849 5974 ts_manager.cc:194] Registered new tserver with Master: 5529f743cceb4762a4ee778b8b51f0c9 (127.0.98.1:35091)
I20260929 00:57:09.718031 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:57:09.730170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:09.842057 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:09.842454 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:09.842543 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:09.846776 6170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:09.846865 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:09.846957 6170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:09.852396 6170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:09.853542 6170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:09.854996 6170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:09.865144 6178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:09.865144 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:09.865168 6176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:09.865435 6170 server_base.cc:1034] running on GCE node
I20260929 00:57:09.866080 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:09.866830 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:09.868052 6170 hybrid_clock.cc:648] HybridClock initialized: now 1790643429868033 us; error 34 us; skew 500 ppm
Sep 29 00:57:09 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:09.871173 6170 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:57:09.872093 6170 webserver.cc:460] Webserver started at http://127.0.98.2:41699/ using document root <none> and password file <none>
I20260929 00:57:09.872676 6170 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:09.872802 6170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:09.873044 6170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:09.875245 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "453836764ccb4b008cb16910199c0858"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "bcbbb4caa104f902b72b9756b009df45"
server_key_iv: "1c15fdceacc8dcaf5c37107a3b60409e"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.875838 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "453836764ccb4b008cb16910199c0858"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "bcbbb4caa104f902b72b9756b009df45"
server_key_iv: "1c15fdceacc8dcaf5c37107a3b60409e"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.879884 6170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:57:09.882694 6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.884011 6170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:09.884255 6170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "453836764ccb4b008cb16910199c0858"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-30s6"
server_key: "bcbbb4caa104f902b72b9756b009df45"
server_key_iv: "1c15fdceacc8dcaf5c37107a3b60409e"
server_key_version: "encryptionkey@0"
I20260929 00:57:09.884378 6170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:09.898085 6170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:09.901739 6170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:09.901961 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:09.902627 6170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:09.903492 6170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:09.903563 6170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.903635 6170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:09.903679 6170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.912452 6170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43005
I20260929 00:57:09.912545 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43005 every 8 connection(s)
I20260929 00:57:09.913416 6170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:09.916936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 6170
I20260929 00:57:09.917073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:09.917433 392 external_mini_cluster.cc:1428] Setting key 96919ee08b2ed3289d01bd7c9a23f56f
Sep 29 00:57:09 dist-test-slave-30s6 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:57:09.931165 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:09.915475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48165 (local address 127.0.98.62:33551)
0929 00:57:09.915630 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:09.915636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:09.916654 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 00:57:09.918601 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:09.918605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:09.918666 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:09.918783 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:09.920575 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.921469 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.923326 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.923561 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.923648 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:09.927063 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:09.927070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:09.927074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:09.927111 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:09.928904 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:09.929442 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:09.929445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:09.929446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:09.929519 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:09.929909 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:09.929912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:09.929913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:09.930086 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:09.930159 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:09.930849 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 00:57:09.930978 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:57:09.932091 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:09.915955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.2:48165)
0929 00:57:09.916486 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:09.916533 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:09.918350 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:09.918997 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:09.919010 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:09.919549 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:09.920375 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:09.920399 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.921645 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.921651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:09.923156 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:09.923171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.923739 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.923747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:09.923984 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:09.924771 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:09.924801 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:09.926837 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:09.929022 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:09.929028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:09.929043 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:09.929336 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:09.929599 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:09.929601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:09.929603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:09.929805 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:09.930174 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:09.930221 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:09.930532 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 00:57:09.930778 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 00:57:09.931145 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":136,"threads_started":1}
I20260929 00:57:09.933575 6299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260929 00:57:09.933861 6299 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:09.934465 6299 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260929 00:57:09.935676 5974 ts_manager.cc:194] Registered new tserver with Master: 453836764ccb4b008cb16910199c0858 (127.0.98.2:43005)
I20260929 00:57:09.936327 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:57:09.949659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:10.064208 6304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:10.064503 6304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:10.064548 6304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:10.068786 6304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:10.068866 6304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:10.068928 6304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:10.074537 6304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:10.075651 6304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:10.076653 6304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:10.086388 6310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:10.086407 6309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:10.086388 6312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:10.086445 6304 server_base.cc:1034] running on GCE node
I20260929 00:57:10.087046 6304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:10.087760 6304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:10.088922 6304 hybrid_clock.cc:648] HybridClock initialized: now 1790643430088910 us; error 30 us; skew 500 ppm
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.091707 6304 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:57:10.092618 6304 webserver.cc:460] Webserver started at http://127.0.98.3:41103/ using document root <none> and password file <none>
I20260929 00:57:10.093205 6304 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:10.093268 6304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:10.093497 6304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:10.095665 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "fcc6a267e5b34a6eaced0b38b925e8c1"
format_stamp: "Formatted at 2026-09-29 00:57:10 on dist-test-slave-30s6"
server_key: "54fc6cfff9f6585e32944c810df3ca8b"
server_key_iv: "dcb95393e166fae2d209671dd73be557"
server_key_version: "encryptionkey@0"
I20260929 00:57:10.096254 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fcc6a267e5b34a6eaced0b38b925e8c1"
format_stamp: "Formatted at 2026-09-29 00:57:10 on dist-test-slave-30s6"
server_key: "54fc6cfff9f6585e32944c810df3ca8b"
server_key_iv: "dcb95393e166fae2d209671dd73be557"
server_key_version: "encryptionkey@0"
I20260929 00:57:10.100250 6304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:10.103102 6319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:10.104660 6304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:10.104842 6304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "fcc6a267e5b34a6eaced0b38b925e8c1"
format_stamp: "Formatted at 2026-09-29 00:57:10 on dist-test-slave-30s6"
server_key: "54fc6cfff9f6585e32944c810df3ca8b"
server_key_iv: "dcb95393e166fae2d209671dd73be557"
server_key_version: "encryptionkey@0"
I20260929 00:57:10.104974 6304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:10.131656 6304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:10.135280 6304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:10.135519 6304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:10.136232 6304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:10.137163 6304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:10.137240 6304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:10.137321 6304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:10.137380 6304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:10.147423 6304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38025
I20260929 00:57:10.147445 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38025 every 8 connection(s)
I20260929 00:57:10.148432 6304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:10.156678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 6304
I20260929 00:57:10.156826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:10.157200 392 external_mini_cluster.cc:1428] Setting key 7ed646d5d3dc727418be66ab27d9e0a1
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:10.163357 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.150519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46023 (local address 127.0.98.62:33551)
0929 00:57:10.150727 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.150731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.151510 (+ 779us) server_negotiation.cc:407] Connection header received
0929 00:57:10.152922 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.152925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.152986 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.153071 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.154376 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.154940 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.156135 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.156381 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.156514 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.159347 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:10.159350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:10.159353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:10.159384 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:10.160924 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.161490 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.161493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.161495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.161580 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.162059 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.162062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.162064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.162399 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:10.162489 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:10.163088 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 00:57:10.163204 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:10.164215 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.150857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.3:46023)
0929 00:57:10.151319 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:10.151367 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:10.152673 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:10.153241 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:10.153248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:10.153633 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:10.154150 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:10.154164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.155066 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.155069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:10.156021 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:10.156030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.156556 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.156559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:10.156709 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.157459 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:10.157477 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:10.159152 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:10.161033 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:10.161038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:10.161053 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:10.161351 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:10.161692 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:10.161695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:10.161698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:10.161946 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:10.162525 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:10.162532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:10.162806 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 00:57:10.162978 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 00:57:10.163278 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260929 00:57:10.165485 6433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260929 00:57:10.165726 6433 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:10.166329 6433 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260929 00:57:10.167412 5974 ts_manager.cc:194] Registered new tserver with Master: fcc6a267e5b34a6eaced0b38b925e8c1 (127.0.98.3:38025)
I20260929 00:57:10.168001 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46023
I20260929 00:57:10.171170 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/test-admin.keytab.
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.187100 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 00:57:10.189990 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:10.188608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.0.98.1:35091)
0929 00:57:10.188911 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.188925 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.188948 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:57:10.188999 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.189005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.189188 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.189333 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.189782 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58010: BlockingRecv error: recv got EOF from 127.0.0.1:58010 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
W20260929 00:57:10.190040 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.191159 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.192150 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:57:10.203152 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.193023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.0.98.1:35091)
0929 00:57:10.193141 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.193145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.193307 (+ 162us) server_negotiation.cc:407] Connection header received
0929 00:57:10.193473 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.193475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.193515 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.193599 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.194130 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.195002 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.195891 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.196279 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.196396 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.199015 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:10.199024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:10.199038 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:10.199065 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:10.201051 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.201503 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.201513 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.201514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.201596 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.202057 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.202060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.202062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.202401 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:10.202577 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:10.202768 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 00:57:10.202960 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 00:57:10.207404 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:10.206402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.0.98.1:35091)
0929 00:57:10.206536 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.206539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.206573 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:57:10.206711 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.206713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.206747 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.206816 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.207258 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58026: BlockingRecv error: recv got EOF from 127.0.0.1:58026 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 00:57:10.207494 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.208737 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.209765 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:57:10.219216 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.210818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.0.98.1:35091)
0929 00:57:10.210942 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.210947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.210963 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:57:10.211051 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.211054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.211277 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.211412 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.211932 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.212477 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.213328 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.213567 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.213686 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.216289 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:10.216292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:10.216294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:10.216318 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:10.217548 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.217984 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.217990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.217992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.218073 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.218390 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.218393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.218395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.218567 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:10.218703 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:10.218938 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 00:57:10.219051 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 00:57:10.221896 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:10.221041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.0.98.1:35091)
0929 00:57:10.221178 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.221182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.221242 (+ 60us) server_negotiation.cc:407] Connection header received
0929 00:57:10.221334 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.221336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.221371 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.221435 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.221775 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58034: BlockingRecv error: recv got EOF from 127.0.0.1:58034 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 00:57:10.221976 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643398448434-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.223031 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.223927 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:57:10.232287 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.224695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.0.98.1:35091)
0929 00:57:10.224835 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.224841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.224922 (+ 81us) server_negotiation.cc:407] Connection header received
0929 00:57:10.225011 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.225013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.225048 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.225111 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.225627 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.226106 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.227023 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.227228 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.227327 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.229573 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:10.229576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:10.229578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:10.229599 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:10.230879 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.231281 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.231284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.231285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.231353 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.231571 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.231573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.231574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.231721 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:10.231814 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:10.231977 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 00:57:10.232059 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:57:10.233311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 6036
I20260929 00:57:10.239420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 6170
I20260929 00:57:10.244886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 6304
I20260929 00:57:10.250108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 5942
2026-09-29T00:57:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2233 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[6453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[6453](info): set up 2 sockets
Sep 29 00:57:10 dist-test-slave-30s6 krb5kdc[6453](info): commencing operation
krb5kdc: starting...
W20260929 00:57:11.310683 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:11 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 00:57:11.438174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45915
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:11.548972 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:11.549304 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:11.549377 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:11.553531 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:11.553624 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:11.553663 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:11.553720 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:11.553759 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:11.559044 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:11.560140 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:11.561183 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:11.570082 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:11.570158 6466 server_base.cc:1034] running on GCE node
W20260929 00:57:11.570104 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:11.570135 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:11.570953 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:11.572075 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:11.573297 6466 hybrid_clock.cc:648] HybridClock initialized: now 1790643431573278 us; error 36 us; skew 500 ppm
Sep 29 00:57:11 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:11.575976 6466 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 00:57:11.576879 6466 webserver.cc:460] Webserver started at http://127.0.98.62:35971/ using document root <none> and password file <none>
I20260929 00:57:11.577435 6466 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:11.577524 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:11.577762 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:11.579833 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "bfa9bcd8f5854b3ea41ad73fd185b311"
format_stamp: "Formatted at 2026-09-29 00:57:11 on dist-test-slave-30s6"
server_key: "f2ba41c8b26fd7ae505c0b41f33259c4"
server_key_iv: "8709c331a10a3979981ccd4924f32d71"
server_key_version: "encryptionkey@0"
I20260929 00:57:11.580394 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "bfa9bcd8f5854b3ea41ad73fd185b311"
format_stamp: "Formatted at 2026-09-29 00:57:11 on dist-test-slave-30s6"
server_key: "f2ba41c8b26fd7ae505c0b41f33259c4"
server_key_iv: "8709c331a10a3979981ccd4924f32d71"
server_key_version: "encryptionkey@0"
I20260929 00:57:11.584533 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 00:57:11.586895 6481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:11.588156 6466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:57:11.588296 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "bfa9bcd8f5854b3ea41ad73fd185b311"
format_stamp: "Formatted at 2026-09-29 00:57:11 on dist-test-slave-30s6"
server_key: "f2ba41c8b26fd7ae505c0b41f33259c4"
server_key_iv: "8709c331a10a3979981ccd4924f32d71"
server_key_version: "encryptionkey@0"
I20260929 00:57:11.588408 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:11.622555 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:11.625975 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:11.626245 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:11.633795 6466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45915
I20260929 00:57:11.634080 6533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45915 every 8 connection(s)
I20260929 00:57:11.634925 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:11.637821 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:11.643527 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: Bootstrap starting.
I20260929 00:57:11.644604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 6466
I20260929 00:57:11.644723 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:11.645031 392 external_mini_cluster.cc:1428] Setting key d8906be29845fd847a76216bd91873ee
I20260929 00:57:11.646520 6534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:11.647478 6534 log.cc:824] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:11.649796 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: No bootstrap required, opened a new log
I20260929 00:57:11.653945 6534 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } }
I20260929 00:57:11.654243 6534 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:11.654312 6534 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa9bcd8f5854b3ea41ad73fd185b311, State: Initialized, Role: FOLLOWER
Sep 29 00:57:11 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:11.654764 6534 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } }
I20260929 00:57:11.654968 6534 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:11.655040 6534 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:11.655176 6534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:11.656486 6534 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } }
I20260929 00:57:11.656944 6534 leader_election.cc:302] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfa9bcd8f5854b3ea41ad73fd185b311; no voters:
I20260929 00:57:11.657351 6534 leader_election.cc:288] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:11.657647 6539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:11.657959 6539 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 1 LEADER]: Becoming Leader. State: Replica: bfa9bcd8f5854b3ea41ad73fd185b311, State: Running, Role: LEADER
I20260929 00:57:11.658378 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } }
I20260929 00:57:11.658839 6534 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:11.660276 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfa9bcd8f5854b3ea41ad73fd185b311. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } }
I20260929 00:57:11.660276 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } }
I20260929 00:57:11.660435 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:11.660435 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:11.661404 6549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:11.662533 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:11.646271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.0.98.62:45915)
0929 00:57:11.646791 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:11.646818 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:11.646869 (+ 51us) server_negotiation.cc:407] Connection header received
0929 00:57:11.647771 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:11.647797 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:11.648180 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:11.648580 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:11.649309 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:11.650811 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:11.651716 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:11.652427 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:11.652617 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:11.655472 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:11.655479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:11.655501 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:11.655538 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:11.657900 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:11.658522 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:11.658528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:11.658534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:11.658690 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:11.659047 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:11.659051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:11.659053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:11.659598 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:11.660794 (+ 1196us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:11.661239 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 00:57:11.661595 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":134,"threads_started":1}
I20260929 00:57:11.664579 6549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:11.670143 6549 catalog_manager.cc:1347] Generated new cluster ID: abd1102f78924fdb8fbf08382611a176
I20260929 00:57:11.670280 6549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:11.692421 6549 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:11.693662 6549 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:11.706481 6549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: Generated new TSK 0
I20260929 00:57:11.707325 6549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 00:57:11.748006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45915
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:11.892994 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:11.893294 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:11.893374 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:11.897444 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:11.897538 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:11.897630 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:11.903093 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:11.904238 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:11.905313 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:11.915181 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:11.915354 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:11.915192 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:11.915656 6560 server_base.cc:1034] running on GCE node
I20260929 00:57:11.916091 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:11.916781 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:11.917973 6560 hybrid_clock.cc:648] HybridClock initialized: now 1790643431917931 us; error 71 us; skew 500 ppm
Sep 29 00:57:11 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:11.921298 6560 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 00:57:11.922726 6560 webserver.cc:460] Webserver started at http://127.0.98.1:40101/ using document root <none> and password file <none>
I20260929 00:57:11.923559 6560 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:11.923652 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:11.923915 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:11.927372 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "126acb8caee2443caf765bd68a5bcf62"
format_stamp: "Formatted at 2026-09-29 00:57:11 on dist-test-slave-30s6"
server_key: "f9fe10a36554618e71d86901efdac539"
server_key_iv: "8d482b8c97eff57e6eccc4ad0432ccf9"
server_key_version: "encryptionkey@0"
I20260929 00:57:11.928151 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "126acb8caee2443caf765bd68a5bcf62"
format_stamp: "Formatted at 2026-09-29 00:57:11 on dist-test-slave-30s6"
server_key: "f9fe10a36554618e71d86901efdac539"
server_key_iv: "8d482b8c97eff57e6eccc4ad0432ccf9"
server_key_version: "encryptionkey@0"
I20260929 00:57:11.933925 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 00:57:11.937619 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:11.939577 6560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:11.939714 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "126acb8caee2443caf765bd68a5bcf62"
format_stamp: "Formatted at 2026-09-29 00:57:11 on dist-test-slave-30s6"
server_key: "f9fe10a36554618e71d86901efdac539"
server_key_iv: "8d482b8c97eff57e6eccc4ad0432ccf9"
server_key_version: "encryptionkey@0"
I20260929 00:57:11.939862 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:11.985140 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:11.988652 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:11.988811 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:11.989368 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:11.990384 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:11.990437 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:11.990489 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:11.990505 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:11.999706 6560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40761
I20260929 00:57:11.999733 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40761 every 8 connection(s)
I20260929 00:57:12.000655 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:12.006815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 6560
I20260929 00:57:12.006920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:12.007231 392 external_mini_cluster.cc:1428] Setting key d3d43a894f7e4ba45bf2432bc5f0ef13
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:12.016465 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.002645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38941 (local address 127.0.98.62:45915)
0929 00:57:12.002793 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.002797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.004032 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 00:57:12.005074 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.005078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.005141 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.005266 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.006648 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.007411 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.008936 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.009197 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.009320 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.012062 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.012069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.012076 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.012114 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.014115 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.014692 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.014697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.014700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.014805 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.015153 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.015160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.015164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.015437 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.015560 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.016096 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.016257 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:57:12.017238 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.002925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:38941)
0929 00:57:12.003854 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.003900 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.004842 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.005478 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.005486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.005875 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.006489 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.006504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.007594 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.007598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.008806 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.008816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.009364 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.009368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.009561 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.010334 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.010351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.011867 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.014284 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.014290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.014303 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.014574 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.014907 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.014909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.014911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.015050 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.015549 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.015555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.015821 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.015985 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.016269 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":552,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:57:12.018807 6689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45915
I20260929 00:57:12.019097 6689 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:12.019654 6689 heartbeater.cc:511] Master 127.0.98.62:45915 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260929 00:57:12.021507 6498 ts_manager.cc:194] Registered new tserver with Master: 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761)
I20260929 00:57:12.022745 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 00:57:12.037210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45915
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:12.151928 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:12.152201 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:12.152246 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:12.156278 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:12.156345 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:12.156405 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:12.162041 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:12.163147 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:12.164113 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:12.173731 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.173695 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.173696 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.174012 6694 server_base.cc:1034] running on GCE node
I20260929 00:57:12.174506 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:12.175134 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:12.176324 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790643432176308 us; error 28 us; skew 500 ppm
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:12.179039 6694 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 00:57:12.179893 6694 webserver.cc:460] Webserver started at http://127.0.98.2:36777/ using document root <none> and password file <none>
I20260929 00:57:12.180446 6694 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:12.180536 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:12.180763 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:12.182807 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f94bd11037f41fcbc42aa7e7c501a56"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "9b43ac7f8e41edb53a3fc959972327d1"
server_key_iv: "2c59eeba4651c6d50b2e9246d2c5f486"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.183364 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f94bd11037f41fcbc42aa7e7c501a56"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "9b43ac7f8e41edb53a3fc959972327d1"
server_key_iv: "2c59eeba4651c6d50b2e9246d2c5f486"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.186968 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:12.189275 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.190524 6694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:12.190654 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "1f94bd11037f41fcbc42aa7e7c501a56"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "9b43ac7f8e41edb53a3fc959972327d1"
server_key_iv: "2c59eeba4651c6d50b2e9246d2c5f486"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.190763 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:12.205968 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:12.209103 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:12.209316 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:12.209918 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:12.210808 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:12.210879 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.210953 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:12.211004 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.219702 6694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35887
I20260929 00:57:12.219728 6822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35887 every 8 connection(s)
I20260929 00:57:12.220611 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:12.223598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 6694
I20260929 00:57:12.223701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:12.224007 392 external_mini_cluster.cc:1428] Setting key b1698655a46bc79f1015e373bd090dfb
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:57:12.236696 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.222539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58129 (local address 127.0.98.62:45915)
0929 00:57:12.222686 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.222690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.223695 (+ 1005us) server_negotiation.cc:407] Connection header received
0929 00:57:12.225413 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.225416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.225465 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.225556 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.227237 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.227757 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.229606 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.229826 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.229955 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.232910 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.232914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.232916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.232940 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.234525 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.235007 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.235009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.235010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.235081 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.235479 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.235482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.235483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.235741 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.235839 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.236432 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.236533 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260929 00:57:12.237706 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.222963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:58129)
0929 00:57:12.223487 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.223550 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.225168 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.225741 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.225750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.226318 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.227045 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.227068 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.227897 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.227905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.229466 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.229481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.230004 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.230008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.230246 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.230892 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.230909 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.232729 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.234639 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.234644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.234659 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.234919 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.235201 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.235207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.235211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.235364 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.235843 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.235848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.236186 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.236366 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.236696 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
I20260929 00:57:12.239084 6823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45915
I20260929 00:57:12.239337 6823 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:12.239852 6823 heartbeater.cc:511] Master 127.0.98.62:45915 requested a full tablet report, sending...
I20260929 00:57:12.240947 6498 ts_manager.cc:194] Registered new tserver with Master: 1f94bd11037f41fcbc42aa7e7c501a56 (127.0.98.2:35887)
I20260929 00:57:12.241596 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 00:57:12.254421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45915
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:12.366581 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:12.366843 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:12.366930 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:12.371263 6828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:12.371327 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:12.371418 6828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:12.376729 6828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:12.377808 6828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:12.378916 6828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:57:12.389238 6828 server_base.cc:1034] running on GCE node
W20260929 00:57:12.389256 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.389256 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.389249 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.390149 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:12.390810 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:12.391968 6828 hybrid_clock.cc:648] HybridClock initialized: now 1790643432391944 us; error 55 us; skew 500 ppm
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:12.394883 6828 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 00:57:12.395833 6828 webserver.cc:460] Webserver started at http://127.0.98.3:42705/ using document root <none> and password file <none>
I20260929 00:57:12.396447 6828 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:12.396564 6828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:12.396818 6828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:12.399147 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "7b2561af2bd74f91b039b18986c2ef01"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "256cc96eaf4820dde5e1f2cbf87fb024"
server_key_iv: "7aba516c99a83f4b13c8e33ed48bdb90"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.399803 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7b2561af2bd74f91b039b18986c2ef01"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "256cc96eaf4820dde5e1f2cbf87fb024"
server_key_iv: "7aba516c99a83f4b13c8e33ed48bdb90"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.403882 6828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:12.406297 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.407567 6828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:12.407730 6828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "7b2561af2bd74f91b039b18986c2ef01"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "256cc96eaf4820dde5e1f2cbf87fb024"
server_key_iv: "7aba516c99a83f4b13c8e33ed48bdb90"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.407851 6828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:12.422269 6828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:12.425204 6828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:12.425426 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:12.426101 6828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:12.426976 6828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:12.427050 6828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.427121 6828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:12.427158 6828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.435967 6828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36963
I20260929 00:57:12.435988 6956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36963 every 8 connection(s)
I20260929 00:57:12.436918 6828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:12.440753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 6828
I20260929 00:57:12.440893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:12.441212 392 external_mini_cluster.cc:1428] Setting key 0f46e34485620af7cfcbd8e1d2559a0e
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:12.453158 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.439044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40907 (local address 127.0.98.62:45915)
0929 00:57:12.439209 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.439214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.440086 (+ 872us) server_negotiation.cc:407] Connection header received
0929 00:57:12.441948 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.441954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.442258 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.442378 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.443904 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.444490 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.445755 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.446111 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.446261 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.449205 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.449210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.449213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.449248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.450913 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.451515 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.451520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.451522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.451641 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.452024 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.452029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.452031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.452228 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.452334 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.452862 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.452989 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:57:12.454022 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.439422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:40907)
0929 00:57:12.439917 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.439973 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.441602 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.442584 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.442592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.443116 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.443750 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.443765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.444629 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.444633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.445619 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.445629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.446300 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.446305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.446497 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.447225 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.447243 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.449008 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.451060 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.451067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.451096 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.451403 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.451748 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.451751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.451753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.451924 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.452345 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.452354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.452625 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.452822 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.453189 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
I20260929 00:57:12.455189 6957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45915
I20260929 00:57:12.455441 6957 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:12.455953 6957 heartbeater.cc:511] Master 127.0.98.62:45915 requested a full tablet report, sending...
I20260929 00:57:12.456934 6496 ts_manager.cc:194] Registered new tserver with Master: 7b2561af2bd74f91b039b18986c2ef01 (127.0.98.3:36963)
I20260929 00:57:12.457530 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40907
I20260929 00:57:12.467490 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 00:57:12.493258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38657
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45915,127.0.98.61:38657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:12.607434 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:12.607772 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:12.607852 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:12.612087 6962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:12.612175 6962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:12.612211 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:12.612269 6962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:12.612313 6962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:12.617798 6962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45915,127.0.98.61:38657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
W20260929 00:57:12.618072 6962 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45915,127.0.98.61:38657'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 00:57:12.619056 6962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:12.620110 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:12.629225 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.629276 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.629393 6962 server_base.cc:1034] running on GCE node
W20260929 00:57:12.629227 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.630095 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:12.631465 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:12.632638 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790643432632607 us; error 49 us; skew 500 ppm
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:12.635727 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 00:57:12.636754 6962 webserver.cc:460] Webserver started at http://127.0.98.61:42369/ using document root <none> and password file <none>
I20260929 00:57:12.637328 6962 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:12.637420 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:12.637648 6962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:12.639709 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data/instance:
uuid: "dc364728f1d540e099d9e2a51aae1e18"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "d04884b3ec6daf454c670493a69f83b3"
server_key_iv: "48f6259ca08873c10444f2ebfd17315a"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.640297 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal/instance:
uuid: "dc364728f1d540e099d9e2a51aae1e18"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "d04884b3ec6daf454c670493a69f83b3"
server_key_iv: "48f6259ca08873c10444f2ebfd17315a"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.644353 6962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 00:57:12.647069 6977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.648375 6962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:12.648514 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
uuid: "dc364728f1d540e099d9e2a51aae1e18"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "d04884b3ec6daf454c670493a69f83b3"
server_key_iv: "48f6259ca08873c10444f2ebfd17315a"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.648634 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:12.674368 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:12.677453 6962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:12.677667 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:12.684660 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38657
I20260929 00:57:12.684976 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38657 every 8 connection(s)
I20260929 00:57:12.685626 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
I20260929 00:57:12.688568 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:12.689858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 6962
I20260929 00:57:12.693415 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:12.705132 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.695544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.0.98.62:45915)
0929 00:57:12.695700 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.695705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.696078 (+ 373us) server_negotiation.cc:407] Connection header received
0929 00:57:12.696217 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.696219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.696258 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.696364 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.697017 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.697567 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.698825 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.699251 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.699371 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.701701 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.701706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.701710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.701743 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.703311 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.703760 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.703763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.703764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.703840 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.704130 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.704133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.704135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.704437 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.704711 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.704908 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.705005 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:57:12.705700 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.691426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.98.61:38657)
0929 00:57:12.691881 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.691901 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.691949 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:57:12.692920 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.692944 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.693384 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.693791 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.694655 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.696206 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.697132 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.697659 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.697834 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.700541 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.700549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.700566 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.700599 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.702467 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.702901 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.702913 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.702917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.703019 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.703303 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.703308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.703309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.703854 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.704058 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.704370 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.704661 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":157,"threads_started":1}
I20260929 00:57:12.705700 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.695501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:48618)
0929 00:57:12.695927 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.695978 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.696116 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.696416 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.696421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.696771 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.696893 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.696899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.697698 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.697704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.698699 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.698709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.699398 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.699400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.699481 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.700049 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.700094 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.701555 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.703407 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.703414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.703420 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.703664 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.703932 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.703937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.703943 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.704030 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.704590 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.704595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.704716 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.704849 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.705006 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":162,"threads_started":1}
I20260929 00:57:12.707216 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 00:57:12.715654 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.707750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.0.98.61:38657)
0929 00:57:12.707858 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.707861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.707916 (+ 55us) server_negotiation.cc:407] Connection header received
0929 00:57:12.708029 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.708034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.708079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.708165 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.708531 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.709345 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.710119 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.710410 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.710528 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.712710 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.712713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.712716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.712743 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.714052 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.714534 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.714538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.714540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.714611 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.714881 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.714884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.714885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.715048 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.715152 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.715378 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.715530 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":40}
I20260929 00:57:12.715654 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.707679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.0.1:49004)
0929 00:57:12.707842 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.707861 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.707939 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.708203 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.708206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.708340 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.708434 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.708439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.709457 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.709459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.710019 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.710028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.710650 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.710656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.710774 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.711319 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.711334 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.712578 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.714202 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.714207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.714210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.714391 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.714703 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.714705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.714707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.714781 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.715142 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.715145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.715221 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.715337 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.715476 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 00:57:12.718571 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Bootstrap starting.
I20260929 00:57:12.720752 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:12.721418 7030 log.cc:824] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:12.723301 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: No bootstrap required, opened a new log
I20260929 00:57:12.726316 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } }
I20260929 00:57:12.726552 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:12.726595 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc364728f1d540e099d9e2a51aae1e18, State: Initialized, Role: FOLLOWER
I20260929 00:57:12.726958 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } }
I20260929 00:57:12.727367 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } } }
I20260929 00:57:12.727478 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:12.728062 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:57:12.732529 7047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:57:12.732612 7047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:57:12.734171 6962 master_runner.cc:428] Detected that this master dc364728f1d540e099d9e2a51aae1e18 is joining an existing cluster
I20260929 00:57:12.734277 6962 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:38657
I20260929 00:57:12.744483 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.736301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.0.98.61:38657)
0929 00:57:12.736438 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.736443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.736712 (+ 269us) server_negotiation.cc:407] Connection header received
0929 00:57:12.736851 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.736854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.736887 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.736976 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.737483 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.738275 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.739423 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.739758 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.739863 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.740938 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.740942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.740944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.740966 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.742729 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.743326 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.743331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.743336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.743400 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.743693 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.743699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.743701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.743861 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.743964 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.744219 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.744332 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":74}
I20260929 00:57:12.744886 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.736232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.0.98.62:45915)
0929 00:57:12.736354 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.736359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.736647 (+ 288us) server_negotiation.cc:407] Connection header received
0929 00:57:12.736832 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.736836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.736892 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.736979 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.737627 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.738412 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.739234 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.739508 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.739602 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.741011 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.741016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.741019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.741043 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.742804 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.743286 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.743291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.743293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.743380 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.744008 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.744013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.744016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.744240 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.744331 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.744621 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.744734 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:57:12.745172 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.736403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.0.1:49006)
0929 00:57:12.736645 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.736656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.736748 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.736985 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.736987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.737261 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.737358 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.737363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.738383 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.738388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.739308 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.739319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.739862 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.739865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.739967 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.740463 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.740473 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.740813 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.742848 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.742852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.742854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.743212 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.743495 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.743498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.743500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.743602 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.743953 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.743956 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.744052 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.744747 (+ 695us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.744873 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
I20260929 00:57:12.745249 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.736184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:48622)
0929 00:57:12.736544 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.736560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.736673 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.737096 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.737099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.737387 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.737508 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.737516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.738510 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.738516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.739107 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.739114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.739630 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.739634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.739754 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.740309 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.740328 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.740851 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.742890 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.742897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.742899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.743188 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.743478 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.743481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.743483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.743588 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.744340 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.744343 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.744430 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.744538 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.744948 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":22,"thread_start_us":95,"threads_started":1}
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 00:57:12.758112 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.749505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.0.1:49022)
0929 00:57:12.749890 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.749929 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.750082 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.750411 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.750417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.750776 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.750876 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.750881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.751594 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.751597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.752368 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.752377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.752820 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.752823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.752927 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.753447 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.753459 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.754896 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.756499 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.756509 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.756515 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.756748 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.757035 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.757038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.757040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.757126 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.757491 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.757499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.757623 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.757763 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.757950 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":80,"threads_started":1}
I20260929 00:57:12.758111 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.749560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.0.98.61:38657)
0929 00:57:12.749680 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.749684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.750006 (+ 322us) server_negotiation.cc:407] Connection header received
0929 00:57:12.750238 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.750240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.750294 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.750379 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.750993 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.751479 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.752480 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.752708 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.752836 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.755029 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.755033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.755035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.755059 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.756388 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.756848 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.756854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.756856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.756938 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.757214 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.757219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.757221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.757388 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.757490 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.757829 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.757938 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:57:12.759721 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:38657: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 38657
}
attrs {
promote: true
}
}
dest_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311"
cas_config_opid_index: -1
I20260929 00:57:12.760493 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } }
I20260929 00:57:12.762360 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dc364728f1d540e099d9e2a51aae1e18 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } }
I20260929 00:57:12.763146 7004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:12.764140 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:38657
I20260929 00:57:12.764544 7004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Refusing update from remote peer bfa9bcd8f5854b3ea41ad73fd185b311: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 00:57:12.764638 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } }
I20260929 00:57:12.764729 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:12.764925 6962 master.cc:537] Master@127.0.98.61:38657 shutting down...
I20260929 00:57:12.765453 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 00:57:12.767364 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 00:57:12.768225 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 -> Peer dc364728f1d540e099d9e2a51aae1e18 (127.0.98.61:38657): Couldn't send request to peer dc364728f1d540e099d9e2a51aae1e18. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 00:57:12.769219 6962 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:12.769488 6962 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:12.769613 6962 tablet_replica.cc:330] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260929 00:57:12.785840 6962 master.cc:552] Master@127.0.98.61:38657 shutdown complete.
I20260929 00:57:12.785931 6962 master_runner.cc:305] Clearing existing system tablet
I20260929 00:57:12.786244 6962 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:12.786307 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:12.787911 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 00:57:12.788962 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.789264 6962 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:12.789404 6962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 00:57:12.789450 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
uuid: "dc364728f1d540e099d9e2a51aae1e18"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "d04884b3ec6daf454c670493a69f83b3"
server_key_iv: "48f6259ca08873c10444f2ebfd17315a"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 00:57:12.791867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41431
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45915,127.0.98.61:38657,127.0.98.60:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:12.792044 6962 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:12.794727 6962 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 00:57:12.794828 6962 log.cc:1196] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 00:57:12.795069 6962 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Deleting consensus metadata
I20260929 00:57:12.795255 6962 master_runner.cc:315] Copying system tablet from 127.0.98.62:45915
I20260929 00:57:12.796342 6962 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45915
I20260929 00:57:12.804287 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.796883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:48628)
0929 00:57:12.797197 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.797209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.797337 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.797602 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.797605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.797817 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.797909 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.797914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.798739 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.798745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.799331 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.799339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.799810 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.799814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.799907 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.800468 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.800489 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.800805 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.802755 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.802758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.802761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.802975 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.803271 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.803276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.803278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.803359 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.803737 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.803740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.803838 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.803943 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.804089 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":91,"threads_started":1}
I20260929 00:57:12.804279 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.796953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.0.98.62:45915)
0929 00:57:12.797084 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.797089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.797290 (+ 201us) server_negotiation.cc:407] Connection header received
0929 00:57:12.797431 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.797434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.797487 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.797576 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.798026 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.798631 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.799453 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.799693 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.799803 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.800952 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.800955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.800958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.800992 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.802625 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.803086 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.803090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.803092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.803179 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.803456 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.803458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.803460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.803643 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.803748 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.803998 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.804115 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:57:12.805169 6518 tablet_copy_service.cc:140] P bfa9bcd8f5854b3ea41ad73fd185b311: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dc364728f1d540e099d9e2a51aae1e18 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48628)
I20260929 00:57:12.805351 6518 tablet_copy_service.cc:161] P bfa9bcd8f5854b3ea41ad73fd185b311: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dc364728f1d540e099d9e2a51aae1e18 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48628: session id = dc364728f1d540e099d9e2a51aae1e18-00000000000000000000000000000000
I20260929 00:57:12.806898 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 00:57:12.808337 6962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 00:57:12.810160 6962 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: tablet copy: Starting download of 0 data blocks...
I20260929 00:57:12.810446 6962 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: tablet copy: Starting download of 1 WAL segments...
I20260929 00:57:12.812062 6962 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 00:57:12.813688 6518 tablet_copy_service.cc:342] P bfa9bcd8f5854b3ea41ad73fd185b311: Request end of tablet copy session dc364728f1d540e099d9e2a51aae1e18-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48628
I20260929 00:57:12.813807 6518 tablet_copy_service.cc:433] P bfa9bcd8f5854b3ea41ad73fd185b311: ending tablet copy session dc364728f1d540e099d9e2a51aae1e18-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dc364728f1d540e099d9e2a51aae1e18
I20260929 00:57:12.820683 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:12.822350 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.822417 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.822620 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.826390 6962 server_base.cc:1034] running on GCE node
I20260929 00:57:12.826639 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:12.826983 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:12.828147 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790643432828130 us; error 38 us; skew 500 ppm
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:12.829702 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 00:57:12.830220 6962 webserver.cc:460] Webserver started at http://127.0.98.61:38433/ using document root <none> and password file <none>
I20260929 00:57:12.830415 6962 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:12.830477 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:12.831625 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:12.832571 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.832813 6962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:12.832907 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
uuid: "dc364728f1d540e099d9e2a51aae1e18"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "d04884b3ec6daf454c670493a69f83b3"
server_key_iv: "48f6259ca08873c10444f2ebfd17315a"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.832991 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:12.864171 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:12.866295 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:12.870343 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38657
I20260929 00:57:12.870958 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
I20260929 00:57:12.876272 7135 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:57:12.876518 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38657 every 8 connection(s)
I20260929 00:57:12.877269 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Bootstrap starting.
I20260929 00:57:12.885241 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:57:12.885558 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: Bootstrap complete.
I20260929 00:57:12.886047 7135 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } }
I20260929 00:57:12.886185 7135 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc364728f1d540e099d9e2a51aae1e18, State: Initialized, Role: LEARNER
I20260929 00:57:12.886322 7135 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } }
I20260929 00:57:12.886674 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } }
I20260929 00:57:12.886835 7135 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:12.886956 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:57:12.889684 7149 catalog_manager.cc:1259] Loaded cluster ID: abd1102f78924fdb8fbf08382611a176
I20260929 00:57:12.889786 7149 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: loading cluster ID for follower catalog manager: success
I20260929 00:57:12.891512 7149 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: acquiring CA information for follower catalog manager: success
I20260929 00:57:12.892709 7149 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 00:57:12.944708 7065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:12.944974 7065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:12.945014 7065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:12.949031 7065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:12.949088 7065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:12.949108 7065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:12.949126 7065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:12.949142 7065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:12.954376 7065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45915,127.0.98.61:38657,127.0.98.60:41431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:12.955401 7065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:12.956382 7065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:12.965157 7155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.965221 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.965303 7065 server_base.cc:1034] running on GCE node
W20260929 00:57:12.965148 7154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.965940 7065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:12.967206 7065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:12.968415 7065 hybrid_clock.cc:648] HybridClock initialized: now 1790643432968378 us; error 46 us; skew 500 ppm
Sep 29 00:57:12 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:12.971313 7065 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 00:57:12.972214 7065 webserver.cc:460] Webserver started at http://127.0.98.60:36459/ using document root <none> and password file <none>
I20260929 00:57:12.972769 7065 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:12.972828 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:12.973053 7065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:12.975150 7065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data/instance:
uuid: "4fb28318fdca4d3dae6657bfbded69cc"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "5f70f3ede403fffc153dc185da03cb12"
server_key_iv: "b93ed7b8e7a5d44035185d150ed1463b"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.975708 7065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal/instance:
uuid: "4fb28318fdca4d3dae6657bfbded69cc"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "5f70f3ede403fffc153dc185da03cb12"
server_key_iv: "b93ed7b8e7a5d44035185d150ed1463b"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.979363 7065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:12.981623 7164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.982885 7065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:12.983009 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
uuid: "4fb28318fdca4d3dae6657bfbded69cc"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "5f70f3ede403fffc153dc185da03cb12"
server_key_iv: "b93ed7b8e7a5d44035185d150ed1463b"
server_key_version: "encryptionkey@0"
I20260929 00:57:12.983111 7065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:13.023626 7065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:13.025169 6689 heartbeater.cc:503] Master 127.0.98.62:45915 was elected leader, sending a full tablet report...
I20260929 00:57:13.027263 7065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:13.027415 7065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:13.035419 7065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41431
I20260929 00:57:13.035697 7216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41431 every 8 connection(s)
I20260929 00:57:13.036464 7065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
I20260929 00:57:13.039575 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:13.039973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 7065
I20260929 00:57:13.044734 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:13.056226 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.041415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.0.98.60:41431)
0929 00:57:13.041806 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.041820 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.041856 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:57:13.042601 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.042628 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.042970 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.043324 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.044155 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.045185 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.046120 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.046756 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.046949 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.049712 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.049717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.049726 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.049754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.051881 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.052407 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.052414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.052420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.052553 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.053831 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.053836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.053839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.054333 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.054519 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.054858 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.055165 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":128,"threads_started":1}
I20260929 00:57:13.056275 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.046708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.0.98.62:45915)
0929 00:57:13.046826 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.046830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.047083 (+ 253us) server_negotiation.cc:407] Connection header received
0929 00:57:13.047354 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.047357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.047404 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.047501 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.048179 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.048830 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.049824 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.050154 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.050304 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.052695 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.052719 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.052723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.052755 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.054601 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.055151 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.055154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.055155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.055230 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.055505 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.055507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.055508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.055671 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.055745 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.056071 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.056167 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:57:13.056403 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.046540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:48640)
0929 00:57:13.046963 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.047012 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.047200 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.047562 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.047568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.047922 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.048044 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.048049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.048984 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.048986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.049709 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.049719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.050307 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.050313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.050417 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.050934 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.050953 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.052574 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.054716 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.054722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.054728 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.054984 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.055319 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.055321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.055323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.055417 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.055767 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.055773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.055885 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.056026 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.056201 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":132,"threads_started":1}
I20260929 00:57:13.057519 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 00:57:13.065968 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.058041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.0.1:49026)
0929 00:57:13.058172 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.058208 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.058287 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.058695 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.058698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.058826 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.058949 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.058957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.060162 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.060164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.060700 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.060707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.061141 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.061144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.061227 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.061660 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.061671 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.062911 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.064720 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.064723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.064724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.064899 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.065123 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.065125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.065127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.065192 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.065601 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.065603 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.065676 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.065752 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.065850 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:57:13.066109 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.058155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.0.98.61:38657)
0929 00:57:13.058476 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.058481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.058495 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:57:13.058538 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.058543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.058594 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.058681 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.059238 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.060032 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.060799 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.061016 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.061132 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.063035 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.063038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.063040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.063064 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.064627 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.064976 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.064979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.064980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.065051 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.065298 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.065301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.065302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.065509 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.065603 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.065814 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.065925 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
I20260929 00:57:13.066813 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 00:57:13.074440 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.067289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41431 (local address 127.0.0.1:60522)
0929 00:57:13.067400 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.067416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.067486 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.067785 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.067787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.067941 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.067997 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.068002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.069038 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.069042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.069533 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.069540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.069984 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.069987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.070060 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.070497 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.070507 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.071651 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.073043 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.073049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.073052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.073230 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.073491 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.073493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.073495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.073582 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.073926 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.073929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.074004 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.074126 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.074290 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 00:57:13.074577 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.067505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.0.98.60:41431)
0929 00:57:13.067630 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.067634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.067643 (+ 9us) server_negotiation.cc:407] Connection header received
0929 00:57:13.067665 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.067667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.067695 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.067776 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.068112 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.068928 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.069650 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.069884 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.069988 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.071777 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.071780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.071783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.071810 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.072939 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.073321 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.073327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.073329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.073399 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.073673 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.073676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.073678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.073834 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.073937 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.074293 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.074407 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:13.077265 7217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Bootstrap starting.
I20260929 00:57:13.079547 7217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:13.080253 7217 log.cc:824] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:13.081998 7217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: No bootstrap required, opened a new log
I20260929 00:57:13.084864 7217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } }
I20260929 00:57:13.085052 7217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:13.085139 7217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb28318fdca4d3dae6657bfbded69cc, State: Initialized, Role: FOLLOWER
I20260929 00:57:13.085541 7217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } }
I20260929 00:57:13.085978 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } } }
I20260929 00:57:13.086109 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:13.086753 7217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:57:13.091547 7234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:57:13.091636 7234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:57:13.094877 7065 master_runner.cc:428] Detected that this master 4fb28318fdca4d3dae6657bfbded69cc is joining an existing cluster
I20260929 00:57:13.094938 7065 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41431
I20260929 00:57:13.105980 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.097207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.0.1:49036)
0929 00:57:13.097667 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.097682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.097808 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.098205 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.098209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.098473 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.098665 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.098674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.099759 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.099764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.100834 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.100844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.101477 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.101480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.101675 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.102351 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.102363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.102709 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.104608 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.104610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.104612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.104797 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.105077 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.105079 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.105081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.105153 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.105555 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.105557 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.105650 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.105729 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.105833 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":349,"spinlock_wait_cycles":4480,"thread_start_us":112,"threads_started":1}
I20260929 00:57:13.106083 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.097338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.0.98.61:38657)
0929 00:57:13.097466 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.097492 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.097746 (+ 254us) server_negotiation.cc:407] Connection header received
0929 00:57:13.097920 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.097924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.097981 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.098034 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.098803 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.099630 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.100954 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.101291 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.101351 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.102863 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.102867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.102871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.102902 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.104480 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.104892 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.104896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.104898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.104989 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.105237 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.105240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.105242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.105463 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.105564 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.105806 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.105933 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 00:57:13.109717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 6560
I20260929 00:57:13.117467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40761
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45915
--webserver_port=40101
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657,127.0.98.60:41431 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:13.120038 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.097339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.0.98.62:45915)
0929 00:57:13.097586 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.097591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.097704 (+ 113us) server_negotiation.cc:407] Connection header received
0929 00:57:13.097876 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.097879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.097931 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.097981 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.098723 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.099552 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.101072 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.101419 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.101495 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.116124 (+ 14629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.116130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.116133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.116163 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.118204 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.118695 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.118699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.118700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.118788 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.119087 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.119090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.119092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.119328 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.119397 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.119743 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.119868 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260929 00:57:13.120319 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.097722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.0.98.60:41431)
0929 00:57:13.097813 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.097817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.098249 (+ 432us) server_negotiation.cc:407] Connection header received
0929 00:57:13.098383 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.098388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.098428 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.098471 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.099005 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.099705 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.100921 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.101236 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.101301 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.116139 (+ 14838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.116146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.116152 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.116180 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.118086 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.119001 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.119004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.119006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.119080 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.119320 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.119323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.119324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.119584 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.119664 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.119979 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.120110 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260929 00:57:13.120570 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.097207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:48644)
0929 00:57:13.097626 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.097640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.097770 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.098130 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.098133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.098414 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.098584 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.098593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.099698 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.099702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.100910 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.100923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.101651 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.101655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.101785 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.115279 (+ 13494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.115297 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.115905 (+ 608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.118339 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.118343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.118345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.118579 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.118904 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.118907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.118909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.118990 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.119469 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.119472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.119564 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.120254 (+ 690us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.120400 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":186,"thread_start_us":236,"threads_started":1}
I20260929 00:57:13.121377 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.097896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41431 (local address 127.0.0.1:60536)
0929 00:57:13.098159 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.098171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.098296 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.098616 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.098621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.098819 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.098907 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.098913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.099809 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.099812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.100799 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.100809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.101462 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.101465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.101714 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.115351 (+ 13637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.115372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.115975 (+ 603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.118606 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.118612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.118617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.118893 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.119169 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.119172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.119173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.119244 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.119707 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.119709 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.119801 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.121080 (+ 1279us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.121232 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":190,"thread_start_us":70,"threads_started":1}
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 00:57:13.134629 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.124387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41431 (local address 127.0.0.1:60552)
0929 00:57:13.124650 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.124667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.124753 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.125340 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.125347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.125561 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.125696 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.125705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.126742 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.126748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.127772 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.127786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.128373 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.128379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.128509 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.129048 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.129069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.130883 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.132903 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.132906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.132909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.133167 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.133460 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.133463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.133466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.133544 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.134038 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.134041 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.134173 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.134299 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.134447 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260929 00:57:13.135237 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.124908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.0.98.60:41431)
0929 00:57:13.125041 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.125046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.125059 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:57:13.125124 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.125127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.125176 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.125256 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.125823 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.126622 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.127917 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.128248 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.128350 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.131047 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.131051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.131054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.131082 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.132781 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.133281 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.133284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.133286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.133366 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.133633 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.133636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.133638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.133908 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.133996 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.134964 (+ 968us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.135079 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:57:13.136181 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41431: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41431
}
attrs {
promote: true
}
}
dest_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311"
cas_config_opid_index: 5
I20260929 00:57:13.136798 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } }
I20260929 00:57:13.138149 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.139791 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41431
I20260929 00:57:13.140026 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.140144 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:13.141063 7190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:13.142545 7190 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Refusing update from remote peer bfa9bcd8f5854b3ea41ad73fd185b311: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 00:57:13.143719 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:13.144477 7065 master.cc:537] Master@127.0.98.60:41431 shutting down...
I20260929 00:57:13.144865 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.139458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.0.98.61:38657)
0929 00:57:13.139584 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.139590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.139736 (+ 146us) server_negotiation.cc:407] Connection header received
0929 00:57:13.139976 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.139980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.140041 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.140114 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.140621 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.141417 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.143176 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.143829 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.143927 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.144092 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.144655 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.144735 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:57:13.145385 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.139227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.0.1:49038)
0929 00:57:13.139670 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.139684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.139799 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.140349 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.140353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.140370 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.140471 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.140476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.141611 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.141614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.143031 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.143043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.144331 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.144334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.144440 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.144486 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.145193 (+ 707us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.145272 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379}
I20260929 00:57:13.145999 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 LEARNER]: Refusing update from remote peer bfa9bcd8f5854b3ea41ad73fd185b311: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 00:57:13.146512 7244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 00:57:13.148432 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
W20260929 00:57:13.149737 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 -> Peer 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60:41431): Couldn't send request to peer 4fb28318fdca4d3dae6657bfbded69cc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 00:57:13.150520 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfa9bcd8f5854b3ea41ad73fd185b311. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.150671 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:57:13.150755 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: true } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.150820 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:57:13.152361 7059 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: attempting to promote NON_VOTER dc364728f1d540e099d9e2a51aae1e18 to VOTER
I20260929 00:57:13.152803 7059 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } }
I20260929 00:57:13.154048 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 LEARNER]: Refusing update from remote peer bfa9bcd8f5854b3ea41ad73fd185b311: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 00:57:13.154567 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 00:57:13.154953 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 -> Peer 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60:41431): Couldn't send request to peer 4fb28318fdca4d3dae6657bfbded69cc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 00:57:13.156486 7065 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:13.156424 7247 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dc364728f1d540e099d9e2a51aae1e18 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.156777 7065 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:13.156708 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dc364728f1d540e099d9e2a51aae1e18 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.156877 7065 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: stopping tablet replica
I20260929 00:57:13.157984 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.158077 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:13.158291 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.158402 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:13.158718 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.158890 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:13.162398 7065 master.cc:552] Master@127.0.98.60:41431 shutdown complete.
I20260929 00:57:13.162458 7065 master_runner.cc:305] Clearing existing system tablet
I20260929 00:57:13.162693 7065 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:13.162753 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:13.164040 7065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 00:57:13.164924 7249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.165187 7065 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:13.165477 7065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:13.165560 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
uuid: "4fb28318fdca4d3dae6657bfbded69cc"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "5f70f3ede403fffc153dc185da03cb12"
server_key_iv: "b93ed7b8e7a5d44035185d150ed1463b"
server_key_version: "encryptionkey@0"
I20260929 00:57:13.167656 7065 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:13.170270 7065 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 00:57:13.170440 7065 log.cc:1196] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 00:57:13.170815 7065 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Deleting consensus metadata
I20260929 00:57:13.171015 7065 master_runner.cc:315] Copying system tablet from 127.0.98.62:45915
I20260929 00:57:13.172231 7065 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45915
I20260929 00:57:13.181394 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.172932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.0.98.62:45915)
0929 00:57:13.173064 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.173069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.173202 (+ 133us) server_negotiation.cc:407] Connection header received
0929 00:57:13.173317 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.173321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.173372 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.173471 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.173990 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.174805 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.175992 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.176324 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.176431 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.177736 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.177740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.177743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.177772 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.179653 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.180146 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.180150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.180152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.180246 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.180536 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.180539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.180541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.180755 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.180854 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.181133 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.181247 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:57:13.181411 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.172814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:48658)
0929 00:57:13.173100 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.173115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.173230 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.173501 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.173504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.173742 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.173856 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.173863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.174946 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.174951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.175864 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.175874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.176449 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.176453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.176550 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.177131 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.177146 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.177572 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.179778 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.179781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.179784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.180032 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.180345 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.180348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.180350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.180442 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.180855 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.180858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.180956 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.181061 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.181203 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260929 00:57:13.182152 6518 tablet_copy_service.cc:140] P bfa9bcd8f5854b3ea41ad73fd185b311: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4fb28318fdca4d3dae6657bfbded69cc ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48658)
I20260929 00:57:13.182308 6518 tablet_copy_service.cc:161] P bfa9bcd8f5854b3ea41ad73fd185b311: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4fb28318fdca4d3dae6657bfbded69cc at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48658: session id = 4fb28318fdca4d3dae6657bfbded69cc-00000000000000000000000000000000
I20260929 00:57:13.183359 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 00:57:13.184573 7065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 00:57:13.187219 7065 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: tablet copy: Starting download of 0 data blocks...
I20260929 00:57:13.187474 7065 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: tablet copy: Starting download of 1 WAL segments...
I20260929 00:57:13.189072 7065 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 00:57:13.190976 6518 tablet_copy_service.cc:342] P bfa9bcd8f5854b3ea41ad73fd185b311: Request end of tablet copy session 4fb28318fdca4d3dae6657bfbded69cc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48658
I20260929 00:57:13.191061 6518 tablet_copy_service.cc:433] P bfa9bcd8f5854b3ea41ad73fd185b311: ending tablet copy session 4fb28318fdca4d3dae6657bfbded69cc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4fb28318fdca4d3dae6657bfbded69cc
I20260929 00:57:13.198834 7065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:13.200668 7256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:13.203519 7065 server_base.cc:1034] running on GCE node
W20260929 00:57:13.203622 7259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:13.206848 7257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:13.207114 7065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:13.207480 7065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:13.208643 7065 hybrid_clock.cc:648] HybridClock initialized: now 1790643433208621 us; error 47 us; skew 500 ppm
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:13.210340 7065 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 00:57:13.210901 7065 webserver.cc:460] Webserver started at http://127.0.98.60:34397/ using document root <none> and password file <none>
I20260929 00:57:13.211184 7065 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:13.211231 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:13.212760 7065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:13.213824 7266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.214124 7065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:13.214241 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
uuid: "4fb28318fdca4d3dae6657bfbded69cc"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "5f70f3ede403fffc153dc185da03cb12"
server_key_iv: "b93ed7b8e7a5d44035185d150ed1463b"
server_key_version: "encryptionkey@0"
I20260929 00:57:13.214327 7065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:13.244331 6823 heartbeater.cc:503] Master 127.0.98.62:45915 was elected leader, sending a full tablet report...
I20260929 00:57:13.247107 7065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:13.249577 7065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:13.254467 7065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41431
I20260929 00:57:13.254503 7318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41431 every 8 connection(s)
I20260929 00:57:13.255043 7065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
I20260929 00:57:13.257068 7319 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:57:13.258209 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Bootstrap starting.
W20260929 00:57:13.267469 7243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:13.267830 7243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:13.267907 7243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 00:57:13.269796 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:57:13.270113 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Bootstrap complete.
I20260929 00:57:13.270743 7319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } }
I20260929 00:57:13.270937 7319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4fb28318fdca4d3dae6657bfbded69cc, State: Initialized, Role: LEARNER
I20260929 00:57:13.271047 7319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } }
I20260929 00:57:13.271396 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.271494 7319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:13.271602 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: This master's current role is: LEARNER
W20260929 00:57:13.274411 7243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:13.274492 7243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:13.274629 7243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:13.274952 7336 catalog_manager.cc:1259] Loaded cluster ID: abd1102f78924fdb8fbf08382611a176
I20260929 00:57:13.275058 7336 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: loading cluster ID for follower catalog manager: success
I20260929 00:57:13.277105 7336 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: acquiring CA information for follower catalog manager: success
I20260929 00:57:13.278586 7336 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:57:13.280555 7243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40101
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657,127.0.98.60:41431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:13.281626 7243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:13.282734 7243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:13.292860 7339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:13.292878 7243 server_base.cc:1034] running on GCE node
W20260929 00:57:13.292979 7338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:13.292980 7341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:13.293620 7243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:13.294377 7243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:13.295562 7243 hybrid_clock.cc:648] HybridClock initialized: now 1790643433295542 us; error 57 us; skew 500 ppm
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:13.298516 7243 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 00:57:13.299497 7243 webserver.cc:460] Webserver started at http://127.0.98.1:40101/ using document root <none> and password file <none>
I20260929 00:57:13.300078 7243 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:13.300195 7243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:13.304234 7243 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:13.306449 7348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.307895 7243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:13.308069 7243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "126acb8caee2443caf765bd68a5bcf62"
format_stamp: "Formatted at 2026-09-29 00:57:11 on dist-test-slave-30s6"
server_key: "f9fe10a36554618e71d86901efdac539"
server_key_iv: "8d482b8c97eff57e6eccc4ad0432ccf9"
server_key_version: "encryptionkey@0"
I20260929 00:57:13.308669 7243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:13.329856 7243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:13.333674 7243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:13.333949 7243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:13.334713 7243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:13.335676 7243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:13.335758 7243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.335831 7243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:13.335875 7243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.346434 7243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40761
I20260929 00:57:13.346493 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40761 every 8 connection(s)
I20260929 00:57:13.347577 7243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:13.347759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 7243
I20260929 00:57:13.347967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 6694
I20260929 00:57:13.355234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35887
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45915
--webserver_port=36777
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657,127.0.98.60:41431 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:13.368232 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.351885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41073 (local address 127.0.98.62:45915)
0929 00:57:13.352023 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.352029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.352467 (+ 438us) server_negotiation.cc:407] Connection header received
0929 00:57:13.353398 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.353401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.353459 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.353514 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.356196 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.356991 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359323 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.359709 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359813 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.363805 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.363811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.363817 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.363857 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.365830 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366498 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.366505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.366509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.366614 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366976 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.366980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.366982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.367274 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.367372 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.367826 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.367975 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:57:13.368646 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.349696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37219 (local address 127.0.98.60:41431)
0929 00:57:13.350838 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.350853 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.351863 (+ 1010us) server_negotiation.cc:407] Connection header received
0929 00:57:13.353390 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.353397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.353464 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.353529 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.356068 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.357047 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359169 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.359542 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359644 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.363130 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.363134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.363138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.363171 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.365311 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366018 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.366022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.366024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.366133 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366652 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.366659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.366662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.366920 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.367003 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.368340 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.368490 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028,"thread_start_us":101,"threads_started":1}
I20260929 00:57:13.369843 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.351224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42481 (local address 127.0.98.61:38657)
0929 00:57:13.351366 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.351373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.352221 (+ 848us) server_negotiation.cc:407] Connection header received
0929 00:57:13.353397 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.353401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.353459 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.353514 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.356181 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.356991 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359152 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.359454 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359577 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.363754 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.363758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.363760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.363797 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.366147 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366682 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.366685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.366687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.366767 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.367048 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.367052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.367055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.367267 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.367354 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.369595 (+ 2241us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.369710 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:57:13.369849 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.351759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:41073)
0929 00:57:13.352376 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.352387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.353161 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.353965 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.353975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.354740 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.356014 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.356020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.357186 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.357189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.359204 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.359210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.359992 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.360159 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.360914 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.360929 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.363619 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.365982 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.365987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.365990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.366353 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.366743 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.366751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.366846 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.367439 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.367444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.367558 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.368499 (+ 941us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.368690 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":350,"spinlock_wait_cycles":12672,"thread_start_us":91,"threads_started":1}
I20260929 00:57:13.370110 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.350221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41431 (local address 127.0.98.1:37219)
0929 00:57:13.351661 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.351722 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.353124 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.353966 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.353977 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.354702 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.355865 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.355889 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.357185 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.357187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.359013 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.359025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.359775 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.359928 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.360709 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.360727 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.362884 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.365461 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.365469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.365488 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.365882 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.366282 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.366287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.366528 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.367079 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.367087 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.367470 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.368838 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.368975 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"mutex_wait_us":436,"spinlock_wait_cycles":42112,"thread_start_us":713,"threads_started":1}
I20260929 00:57:13.370949 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.350618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.98.1:42481)
0929 00:57:13.352102 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.352126 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.353161 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.353965 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.353975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.354747 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.356010 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.356020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.357186 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.357189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.359019 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.359033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.359631 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.359910 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.360782 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.360802 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.363604 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.366282 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.366287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.366571 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.366864 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.366867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.366869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.366948 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.367395 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.367398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.367471 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.367711 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.368165 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1425,"mutex_wait_us":439,"spinlock_wait_cycles":7040,"thread_start_us":128,"threads_started":1}
I20260929 00:57:13.371383 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41431
I20260929 00:57:13.371410 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45915
I20260929 00:57:13.371631 7463 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.371631 7462 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.371907 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38657
I20260929 00:57:13.371973 7464 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.372247 7462 heartbeater.cc:511] Master 127.0.98.60:41431 requested a full tablet report, sending...
I20260929 00:57:13.372248 7464 heartbeater.cc:511] Master 127.0.98.61:38657 requested a full tablet report, sending...
I20260929 00:57:13.372251 7463 heartbeater.cc:511] Master 127.0.98.62:45915 requested a full tablet report, sending...
I20260929 00:57:13.373817 7283 ts_manager.cc:194] Registered new tserver with Master: 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761)
I20260929 00:57:13.373817 7099 ts_manager.cc:194] Registered new tserver with Master: 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761)
I20260929 00:57:13.374924 6496 ts_manager.cc:194] Re-registered known tserver with Master: 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761)
I20260929 00:57:13.375521 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41073
I20260929 00:57:13.459774 6957 heartbeater.cc:503] Master 127.0.98.62:45915 was elected leader, sending a full tablet report...
W20260929 00:57:13.477715 7470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:13.478036 7470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:13.478116 7470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:13.482200 7470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:13.482295 7470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:13.482384 7470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:13.487736 7470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36777
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657,127.0.98.60:41431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:13.488819 7470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:13.489856 7470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:57:13.499943 7470 server_base.cc:1034] running on GCE node
W20260929 00:57:13.499961 7475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:13.499996 7478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:13.499917 7476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:13.500603 7470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:13.501291 7470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:13.502478 7470 hybrid_clock.cc:648] HybridClock initialized: now 1790643433502457 us; error 44 us; skew 500 ppm
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:13.505398 7470 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 00:57:13.506440 7470 webserver.cc:460] Webserver started at http://127.0.98.2:36777/ using document root <none> and password file <none>
I20260929 00:57:13.507035 7470 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:13.507174 7470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:13.511140 7470 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:57:13.513445 7485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.514849 7470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:13.514969 7470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "1f94bd11037f41fcbc42aa7e7c501a56"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "9b43ac7f8e41edb53a3fc959972327d1"
server_key_iv: "2c59eeba4651c6d50b2e9246d2c5f486"
server_key_version: "encryptionkey@0"
I20260929 00:57:13.515535 7470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:13.534514 7470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:13.537624 7470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:13.537783 7470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:13.538394 7470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:13.539242 7470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:13.539289 7470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.539330 7470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:13.539353 7470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.548942 7470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35887
I20260929 00:57:13.548977 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35887 every 8 connection(s)
I20260929 00:57:13.549981 7470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:13.554673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 7470
I20260929 00:57:13.554868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 6828
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:13.562470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36963
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45915
--webserver_port=42705
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657,127.0.98.60:41431 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 00:57:13.567173 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.551900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60421 (local address 127.0.98.61:38657)
0929 00:57:13.552043 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.552048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.553494 (+ 1446us) server_negotiation.cc:407] Connection header received
0929 00:57:13.554402 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.554406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.554464 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.554518 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.557338 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.558280 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.559833 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.560267 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.560358 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.563106 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.563111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.563119 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.563169 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.564872 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.565344 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.565348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.565350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.565451 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.565783 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.565787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.565788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.566021 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.566118 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.566857 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.566984 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:57:13.567178 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.551974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60801 (local address 127.0.98.62:45915)
0929 00:57:13.552096 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.552100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.553129 (+ 1029us) server_negotiation.cc:407] Connection header received
0929 00:57:13.554402 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.554406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.554464 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.554518 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.557333 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.557869 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.559174 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.559473 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.559617 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.562693 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.562697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.562699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.562729 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.564554 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.565277 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.565282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.565285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.565374 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.565793 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.565797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.565798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.565997 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.566061 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.566857 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.566984 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:57:13.568950 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.552434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.98.2:60421)
0929 00:57:13.553397 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.553411 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.554128 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.554922 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.554931 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.556183 (+ 1252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.557156 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.557179 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.558428 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.558434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.559657 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.559671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.560539 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.560546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.560674 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.561171 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.561191 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.562939 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.564992 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.564996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.564998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.565246 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.565547 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.565549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.565551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.565680 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.566144 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.566151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.566543 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.567383 (+ 840us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.567846 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":313,"spinlock_wait_cycles":49920,"thread_start_us":673,"threads_started":1}
I20260929 00:57:13.568957 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.552434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:60801)
0929 00:57:13.552959 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.553000 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.554128 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.555468 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.555471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.556216 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.557155 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.557177 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.558016 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.558020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.559021 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.559030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.559655 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.559659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.559859 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.560546 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.560563 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.562467 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.564701 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.564708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.564733 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.565145 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.565471 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.565473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.565475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.565687 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.566144 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.566151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.566565 (+ 414us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.567351 (+ 786us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.567802 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":164,"spinlock_wait_cycles":63104,"thread_start_us":155,"threads_started":1}
I20260929 00:57:13.570363 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45915
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 00:57:13.570642 7600 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.571228 7600 heartbeater.cc:511] Master 127.0.98.62:45915 requested a full tablet report, sending...
I20260929 00:57:13.572513 6496 ts_manager.cc:194] Re-registered known tserver with Master: 1f94bd11037f41fcbc42aa7e7c501a56 (127.0.98.2:35887)
I20260929 00:57:13.573252 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60801
I20260929 00:57:13.573563 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38657
I20260929 00:57:13.573660 7601 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.573850 7601 heartbeater.cc:511] Master 127.0.98.61:38657 requested a full tablet report, sending...
I20260929 00:57:13.574589 7099 ts_manager.cc:194] Registered new tserver with Master: 1f94bd11037f41fcbc42aa7e7c501a56 (127.0.98.2:35887)
I20260929 00:57:13.575244 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.553785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55569 (local address 127.0.98.60:41431)
0929 00:57:13.553916 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.553920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.554011 (+ 91us) server_negotiation.cc:407] Connection header received
0929 00:57:13.554374 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.554378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.554438 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.554513 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.564570 (+ 10057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.565388 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.566507 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.566931 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.567038 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.571041 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.571046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.571048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.571087 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.572923 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.573337 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.573343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.573345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.573433 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.573826 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.573829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.573831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.574077 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.574175 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.575020 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.575134 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:57:13.575738 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.553672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41431 (local address 127.0.98.2:55569)
0929 00:57:13.553937 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.553955 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.554136 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.564018 (+ 9882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.564023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.564247 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.564393 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.564401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.565543 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.565547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.566386 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.566395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.567213 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.567626 (+ 413us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.567747 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.568291 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.568311 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.570870 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.573022 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.573024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.573025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.573221 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.573545 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.573548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.573550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.573729 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.574238 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.574241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.574820 (+ 579us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.575457 (+ 637us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.575619 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
I20260929 00:57:13.576840 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41431
I20260929 00:57:13.576941 7599 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.577129 7599 heartbeater.cc:511] Master 127.0.98.60:41431 requested a full tablet report, sending...
I20260929 00:57:13.577927 7283 ts_manager.cc:194] Registered new tserver with Master: 1f94bd11037f41fcbc42aa7e7c501a56 (127.0.98.2:35887)
I20260929 00:57:13.636997 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.632721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41431 (local address 127.0.0.1:60558)
0929 00:57:13.632891 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.632923 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.633055 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.633461 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.633464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.633484 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.633580 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.633584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.634894 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.634900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.635759 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.635769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.636547 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.636551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.636644 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.636668 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.636756 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.636846 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 00:57:13.636997 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.632832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.98.60:41431)
0929 00:57:13.632968 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.632973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.632997 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:57:13.633175 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.633178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.633243 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.633344 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.634050 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.634745 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.635901 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.636395 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.636527 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.636695 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.636773 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.636852 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:57:13.637749 7293 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 00:57:13.638149 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfa9bcd8f5854b3ea41ad73fd185b311. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.638358 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: This master's current role is: LEARNER
W20260929 00:57:13.687083 7606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:13.687453 7606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:13.687501 7606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:13.691850 7606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:13.691907 7606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:13.691999 7606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:13.697611 7606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42705
--tserver_master_addrs=127.0.98.62:45915,127.0.98.61:38657,127.0.98.60:41431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:13.698732 7606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:13.699779 7606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:13.709615 7611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:13.709600 7612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:13.709600 7614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:13.709797 7606 server_base.cc:1034] running on GCE node
I20260929 00:57:13.710311 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:13.710974 7606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:13.712144 7606 hybrid_clock.cc:648] HybridClock initialized: now 1790643433712124 us; error 28 us; skew 500 ppm
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:13.715059 7606 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 00:57:13.715925 7606 webserver.cc:460] Webserver started at http://127.0.98.3:42705/ using document root <none> and password file <none>
I20260929 00:57:13.716442 7606 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:13.716509 7606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:13.720263 7606 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:13.722425 7621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.723659 7606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:13.723790 7606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "7b2561af2bd74f91b039b18986c2ef01"
format_stamp: "Formatted at 2026-09-29 00:57:12 on dist-test-slave-30s6"
server_key: "256cc96eaf4820dde5e1f2cbf87fb024"
server_key_iv: "7aba516c99a83f4b13c8e33ed48bdb90"
server_key_version: "encryptionkey@0"
I20260929 00:57:13.724310 7606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:13.752746 7606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:13.755991 7606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:13.756207 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:13.756819 7606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:13.757692 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:13.757763 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.757836 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:13.757880 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.767849 7606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36963
I20260929 00:57:13.767874 7734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36963 every 8 connection(s)
I20260929 00:57:13.768885 7606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:13.771492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 7606
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 00:57:13 dist-test-slave-30s6 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 00:57:13.788735 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.773872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.0.98.62:45915)
0929 00:57:13.774237 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.774242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.774258 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:57:13.774309 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.774313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.774384 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.774499 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.775423 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.776241 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.777473 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.777895 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.778057 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.780556 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.780560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.780563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.780605 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.782875 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.783380 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.783384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.783386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.783473 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.783795 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.783798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.783800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.784314 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.784423 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.788371 (+ 3948us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.788521 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":263,"spinlock_wait_cycles":4864,"thread_start_us":78,"threads_started":1}
I20260929 00:57:13.790736 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.771006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59161 (local address 127.0.98.61:38657)
0929 00:57:13.771171 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.771177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.773318 (+ 2141us) server_negotiation.cc:407] Connection header received
0929 00:57:13.775178 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.775182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.775242 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.775296 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.778574 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.779336 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.781251 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.781641 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.781751 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.786722 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.786726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.786728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.786775 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.789055 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.789496 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.789502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.789504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.789595 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.789884 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.789887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.789889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.790072 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.790148 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.790490 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.790600 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"spinlock_wait_cycles":11392}
I20260929 00:57:13.791106 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.773518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58155 (local address 127.0.98.60:41431)
0929 00:57:13.773739 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.773744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.773762 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:57:13.775792 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.775797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.775874 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.775945 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.777515 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.778042 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.779823 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.780225 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.780333 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.784963 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.784967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.784970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.784995 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.788134 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.789268 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.789272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.789273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.789360 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.789655 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.789659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.789660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.789880 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.789965 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.790687 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.790971 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260929 00:57:13.791384 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.770884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45101 (local address 127.0.98.62:45915)
0929 00:57:13.771025 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.771029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.773374 (+ 2345us) server_negotiation.cc:407] Connection header received
0929 00:57:13.775292 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.775298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.775379 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.775466 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.777147 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.777987 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.780032 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.780403 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.780517 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.785042 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.785049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.785053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.785074 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.786882 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.787693 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.787713 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.787715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.787802 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.788254 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.788259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.788263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.788514 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.788612 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.791144 (+ 2532us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.791249 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:13.791561 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.772714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41431 (local address 127.0.98.3:58155)
0929 00:57:13.773617 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.773633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.774901 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.776128 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.776131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.777271 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.777420 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.777428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.778190 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.778193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.779683 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.779694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.780523 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.780528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.780766 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.781955 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.781976 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.784792 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.788904 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.788907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.788909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.789160 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.789473 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.789476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.789477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.789559 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.790009 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.790012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.790096 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.790268 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.791367 (+ 1099us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":6,"thread_start_us":507,"threads_started":1}
I20260929 00:57:13.791766 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.771390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.98.3:59161)
0929 00:57:13.772768 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.772784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.774922 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.775649 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.775659 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.777362 (+ 1703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.778424 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.778437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.780178 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.780182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.781111 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.781125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.783547 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.783552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.783661 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.784602 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.784621 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.786564 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.789165 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.789168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.789170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.789382 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.789691 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.789694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.789696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.789771 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.790235 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.790238 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.790318 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.790890 (+ 572us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.791026 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"mutex_wait_us":923,"spinlock_wait_cycles":4864,"thread_start_us":245,"threads_started":1}
I20260929 00:57:13.792933 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38657
I20260929 00:57:13.793179 7737 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.793778 7737 heartbeater.cc:511] Master 127.0.98.61:38657 requested a full tablet report, sending...
I20260929 00:57:13.794294 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41431
I20260929 00:57:13.794394 7735 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.794548 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.771390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:45101)
0929 00:57:13.772656 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.772719 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.774901 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.775943 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.775947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.776298 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.777011 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.777029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.778168 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.778174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.779863 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.779877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.780652 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.780657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.780773 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.781966 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.781984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.784864 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.787062 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.787068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.787103 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.787517 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.787929 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.787933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.787935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.788131 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.788685 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.788692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.789304 (+ 612us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.789523 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.789890 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":613,"thread_start_us":867,"threads_started":1}
I20260929 00:57:13.791970 6496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:57:13.794965 7099 ts_manager.cc:194] Registered new tserver with Master: 7b2561af2bd74f91b039b18986c2ef01 (127.0.98.3:36963)
I20260929 00:57:13.795710 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45915
I20260929 00:57:13.795775 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.775736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.0.98.61:38657)
0929 00:57:13.776312 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.776317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.776776 (+ 459us) server_negotiation.cc:407] Connection header received
0929 00:57:13.776827 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.776832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.776887 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.776978 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.782221 (+ 5243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.783916 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.786167 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.786590 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.786702 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.788704 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.788707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.788710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.788736 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.793658 (+ 4922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.794150 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.794153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.794155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.794258 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.794552 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.794557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.794559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.794756 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.794862 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.795507 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.795623 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":88,"threads_started":1}
I20260929 00:57:13.795794 7736 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.795933 7736 heartbeater.cc:511] Master 127.0.98.62:45915 requested a full tablet report, sending...
W20260929 00:57:13.795984 6496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:13.794575 7735 heartbeater.cc:511] Master 127.0.98.60:41431 requested a full tablet report, sending...
I20260929 00:57:13.796692 6498 ts_manager.cc:194] Re-registered known tserver with Master: 7b2561af2bd74f91b039b18986c2ef01 (127.0.98.3:36963)
I20260929 00:57:13.797271 7283 ts_manager.cc:194] Registered new tserver with Master: 7b2561af2bd74f91b039b18986c2ef01 (127.0.98.3:36963)
I20260929 00:57:13.797348 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45101
I20260929 00:57:13.799240 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: true } } }
I20260929 00:57:13.799393 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:13.801190 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.774149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.0.98.60:41431)
0929 00:57:13.774735 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.774741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.776460 (+ 1719us) server_negotiation.cc:407] Connection header received
0929 00:57:13.776712 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.776719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.776782 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.776866 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.781314 (+ 4448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.782163 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.783949 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.784335 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.784429 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.797479 (+ 13050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.797487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.797491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.797515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.799381 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.799912 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.799915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.799917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.800009 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.800295 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.800298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.800300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.800498 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.800634 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.800864 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.801002 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":105,"threads_started":1}
I20260929 00:57:13.802922 7753 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: attempting to promote NON_VOTER 4fb28318fdca4d3dae6657bfbded69cc to VOTER
I20260929 00:57:13.803264 7753 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } }
I20260929 00:57:13.806207 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Refusing update from remote peer bfa9bcd8f5854b3ea41ad73fd185b311: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 00:57:13.806728 7754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 00:57:13.807000 7292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 LEARNER]: Refusing update from remote peer bfa9bcd8f5854b3ea41ad73fd185b311: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 00:57:13.807521 7754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 00:57:13.808666 7247 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:13.809098 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:13.810153 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:13.810280 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:13.810408 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:13.810493 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:13.811017 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:13.811153 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:13.811615 7753 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:13.811730 7753 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:13.811985 7292 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:13.814807 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:13.814937 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:13.818362 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.812694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35887 (local address 127.0.0.1:60248)
0929 00:57:13.813225 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.813242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.813323 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.814448 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.814452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.814473 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.814578 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.814584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.816116 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.816119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.816861 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.816870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.817941 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.817945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.818046 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.818071 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.818146 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.818234 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457}
I20260929 00:57:13.818506 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.812987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.0.98.2:35887)
0929 00:57:13.813830 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.813843 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.813882 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:57:13.814017 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.814024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.814282 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.814745 (+ 463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.814914 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.815987 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.816995 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.817799 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.817913 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.818053 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.818248 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.818347 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":113,"threads_started":1}
I20260929 00:57:13.820384 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.813926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36963 (local address 127.0.0.1:42982)
0929 00:57:13.815315 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.815335 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.815446 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.815940 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.815943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.815960 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.816073 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.816079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.817249 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.817254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.818391 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.818403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.819950 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.819954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.820038 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.820064 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.820155 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.820245 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":94,"threads_started":1}
I20260929 00:57:13.820412 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.814515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.0.98.3:36963)
0929 00:57:13.814801 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.814813 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.815426 (+ 613us) server_negotiation.cc:407] Connection header received
0929 00:57:13.815564 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.815569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.815789 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.816205 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.816332 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.817116 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.819007 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.819779 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.819893 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.820033 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.820182 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.820272 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":115,"threads_started":1}
I20260929 00:57:13.820614 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.813562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40761 (local address 127.0.0.1:53208)
0929 00:57:13.814215 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.814230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.814318 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.817011 (+ 2693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.817014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.817032 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.817109 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.817114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.818435 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.818438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.819302 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.819332 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.820247 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.820250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.820330 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.820349 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.820410 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.820481 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":138,"threads_started":1}
I20260929 00:57:13.821242 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.813776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.0.98.1:40761)
0929 00:57:13.815678 (+ 1902us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.815693 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.815712 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:57:13.815776 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.815780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.815945 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.816105 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.817234 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.818300 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.819469 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.820127 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.820851 (+ 724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.821002 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.821054 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.821135 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1802,"thread_start_us":89,"threads_started":1}
I20260929 00:57:13.821791 7533 tablet_service.cc:1467] Processing CreateTablet for tablet fbde041396cc4d0382c373f10845a05b (DEFAULT_TABLE table=test-table [id=e18f205744ea402693a0ecedee426993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:13.823089 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbde041396cc4d0382c373f10845a05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:13.823623 7669 tablet_service.cc:1467] Processing CreateTablet for tablet fbde041396cc4d0382c373f10845a05b (DEFAULT_TABLE table=test-table [id=e18f205744ea402693a0ecedee426993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:13.823890 7396 tablet_service.cc:1467] Processing CreateTablet for tablet fbde041396cc4d0382c373f10845a05b (DEFAULT_TABLE table=test-table [id=e18f205744ea402693a0ecedee426993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:13.824525 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbde041396cc4d0382c373f10845a05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:13.824842 7396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbde041396cc4d0382c373f10845a05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:13.831499 7768 tablet_bootstrap.cc:492] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Bootstrap starting.
I20260929 00:57:13.832899 7770 tablet_bootstrap.cc:492] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Bootstrap starting.
I20260929 00:57:13.833464 7769 tablet_bootstrap.cc:492] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Bootstrap starting.
I20260929 00:57:13.833925 7768 tablet_bootstrap.cc:654] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:13.834730 7768 log.cc:824] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:13.835134 7770 tablet_bootstrap.cc:654] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:13.835548 7769 tablet_bootstrap.cc:654] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:13.835970 7770 log.cc:824] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:13.836315 7769 log.cc:824] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:13.836763 7768 tablet_bootstrap.cc:492] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: No bootstrap required, opened a new log
I20260929 00:57:13.836966 7768 ts_tablet_manager.cc:1397] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:57:13.837863 7770 tablet_bootstrap.cc:492] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: No bootstrap required, opened a new log
I20260929 00:57:13.838058 7770 ts_tablet_manager.cc:1397] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 00:57:13.840998 7768 raft_consensus.cc:348] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:13.841218 7768 raft_consensus.cc:374] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:13.841251 7768 raft_consensus.cc:729] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2561af2bd74f91b039b18986c2ef01, State: Initialized, Role: FOLLOWER
I20260929 00:57:13.841691 7768 consensus_queue.cc:260] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:13.841642 7770 raft_consensus.cc:348] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:13.841884 7770 raft_consensus.cc:374] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:13.841938 7770 raft_consensus.cc:729] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f94bd11037f41fcbc42aa7e7c501a56, State: Initialized, Role: FOLLOWER
I20260929 00:57:13.842404 7736 heartbeater.cc:503] Master 127.0.98.62:45915 was elected leader, sending a full tablet report...
I20260929 00:57:13.842435 7770 consensus_queue.cc:260] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:13.842747 7768 ts_tablet_manager.cc:1428] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260929 00:57:13.843485 7770 ts_tablet_manager.cc:1428] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 00:57:13.845499 7600 heartbeater.cc:503] Master 127.0.98.62:45915 was elected leader, sending a full tablet report...
I20260929 00:57:13.849210 7769 tablet_bootstrap.cc:492] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: No bootstrap required, opened a new log
I20260929 00:57:13.849450 7769 ts_tablet_manager.cc:1397] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260929 00:57:13.853329 7769 raft_consensus.cc:348] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:13.853581 7769 raft_consensus.cc:374] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:13.853645 7769 raft_consensus.cc:729] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126acb8caee2443caf765bd68a5bcf62, State: Initialized, Role: FOLLOWER
I20260929 00:57:13.854210 7769 consensus_queue.cc:260] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:13.855044 7463 heartbeater.cc:503] Master 127.0.98.62:45915 was elected leader, sending a full tablet report...
I20260929 00:57:13.856974 7769 ts_tablet_manager.cc:1428] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 00:57:14.020893 7738 tablet.cc:2369] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:57:14.052380 7602 tablet.cc:2369] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:57:14.101343 7465 tablet.cc:2369] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:14.133266 7775 raft_consensus.cc:482] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:14.133617 7775 raft_consensus.cc:504] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:14.135023 7775 leader_election.cc:288] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761), 7b2561af2bd74f91b039b18986c2ef01 (127.0.98.3:36963)
I20260929 00:57:14.139134 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.135307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40761 (local address 127.0.98.2:52319)
0929 00:57:14.135640 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:14.135652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:14.135775 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:14.136161 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:14.136163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:14.136177 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:14.136291 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:14.136300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.137099 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.137101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:14.137909 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:14.137917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.138676 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.138682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:14.138781 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.138811 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 00:57:14.138887 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 00:57:14.138943 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":46,"thread_start_us":72,"threads_started":1}
I20260929 00:57:14.139146 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.135713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52319 (local address 127.0.98.1:40761)
0929 00:57:14.135881 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:14.135886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:14.135901 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:14.135964 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:14.135967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:14.136043 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:14.136126 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:14.136418 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.136981 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.138035 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.138548 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.138643 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.138743 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:14.138926 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 00:57:14.139016 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:57:14.139209 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.135307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36963 (local address 127.0.98.2:55157)
0929 00:57:14.135593 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:14.135612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:14.135753 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:14.136058 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:14.136064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:14.136087 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:14.136207 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:14.136214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.136996 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.137002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:14.138022 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:14.138033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.138753 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.138760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:14.138862 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.138914 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 00:57:14.138998 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 00:57:14.139066 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":35,"thread_start_us":82,"threads_started":1}
I20260929 00:57:14.139231 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.135458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55157 (local address 127.0.98.3:36963)
0929 00:57:14.135593 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:14.135599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:14.135731 (+ 132us) server_negotiation.cc:407] Connection header received
0929 00:57:14.135889 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:14.135893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:14.135952 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:14.136010 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:14.136338 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.136874 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.138151 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.138610 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.138720 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.138784 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:14.139026 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 00:57:14.139102 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:57:14.139803 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbde041396cc4d0382c373f10845a05b" candidate_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b2561af2bd74f91b039b18986c2ef01" is_pre_election: true
I20260929 00:57:14.139957 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbde041396cc4d0382c373f10845a05b" candidate_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126acb8caee2443caf765bd68a5bcf62" is_pre_election: true
I20260929 00:57:14.140146 7689 raft_consensus.cc:2454] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f94bd11037f41fcbc42aa7e7c501a56 in term 0.
I20260929 00:57:14.140300 7416 raft_consensus.cc:2454] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f94bd11037f41fcbc42aa7e7c501a56 in term 0.
I20260929 00:57:14.140699 7486 leader_election.cc:302] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f94bd11037f41fcbc42aa7e7c501a56, 7b2561af2bd74f91b039b18986c2ef01; no voters:
I20260929 00:57:14.140964 7775 raft_consensus.cc:2781] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:14.141075 7775 raft_consensus.cc:482] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:14.141139 7775 raft_consensus.cc:3037] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:14.142242 7775 raft_consensus.cc:504] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:14.142679 7775 leader_election.cc:288] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [CANDIDATE]: Term 1 election: Requested vote from peers 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761), 7b2561af2bd74f91b039b18986c2ef01 (127.0.98.3:36963)
I20260929 00:57:14.143013 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbde041396cc4d0382c373f10845a05b" candidate_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126acb8caee2443caf765bd68a5bcf62"
I20260929 00:57:14.143111 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbde041396cc4d0382c373f10845a05b" candidate_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b2561af2bd74f91b039b18986c2ef01"
I20260929 00:57:14.143187 7416 raft_consensus.cc:3037] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:14.143263 7689 raft_consensus.cc:3037] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:14.144307 7416 raft_consensus.cc:2454] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f94bd11037f41fcbc42aa7e7c501a56 in term 1.
I20260929 00:57:14.144390 7689 raft_consensus.cc:2454] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f94bd11037f41fcbc42aa7e7c501a56 in term 1.
I20260929 00:57:14.144622 7489 leader_election.cc:302] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 126acb8caee2443caf765bd68a5bcf62, 1f94bd11037f41fcbc42aa7e7c501a56; no voters:
I20260929 00:57:14.144797 7775 raft_consensus.cc:2781] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:14.145020 7775 raft_consensus.cc:686] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 LEADER]: Becoming Leader. State: Replica: 1f94bd11037f41fcbc42aa7e7c501a56, State: Running, Role: LEADER
I20260929 00:57:14.145411 7775 consensus_queue.cc:237] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } }
I20260929 00:57:14.148640 6496 catalog_manager.cc:5495] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f94bd11037f41fcbc42aa7e7c501a56 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:14.186790 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.183092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.0.98.2:35887)
0929 00:57:14.183238 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:14.183243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:14.183257 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:57:14.183401 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:14.183403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:14.183458 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:14.183597 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:14.183857 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.184352 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.185207 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.185422 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.185552 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.185880 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:14.186411 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:14.186551 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:14.186585 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 00:57:14.186645 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:57:14.193246 7416 raft_consensus.cc:1261] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 1 FOLLOWER]: Refusing update from remote peer 1f94bd11037f41fcbc42aa7e7c501a56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:14.193246 7689 raft_consensus.cc:1261] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 1 FOLLOWER]: Refusing update from remote peer 1f94bd11037f41fcbc42aa7e7c501a56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:14.193982 7775 consensus_queue.cc:1035] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:14.194204 7779 consensus_queue.cc:1035] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:14.203598 7784 mvcc.cc:205] Tried to move back new op lower bound from 7334475506448547840 to 7334475506262253568. Current Snapshot: MvccSnapshot[applied={T|T < 7334475506448547840}]
I20260929 00:57:14.212605 6496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:14.212796 6496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:14.216929 6496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bfa9bcd8f5854b3ea41ad73fd185b311: Sending DeleteTablet for 3 replicas of tablet fbde041396cc4d0382c373f10845a05b
I20260929 00:57:14.217998 7533 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde041396cc4d0382c373f10845a05b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60248
I20260929 00:57:14.218021 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde041396cc4d0382c373f10845a05b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42982
I20260929 00:57:14.218669 7396 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde041396cc4d0382c373f10845a05b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53208
I20260929 00:57:14.219455 7791 tablet_replica.cc:330] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: stopping tablet replica
I20260929 00:57:14.219767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 6466
I20260929 00:57:14.220834 7792 tablet_replica.cc:330] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: stopping tablet replica
I20260929 00:57:14.221179 7792 raft_consensus.cc:2229] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:14.221575 7792 raft_consensus.cc:2258] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:14.219851 7791 raft_consensus.cc:2229] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:14.222172 7791 raft_consensus.cc:2258] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:14.223883 7791 ts_tablet_manager.cc:1905] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:14.223884 7792 ts_tablet_manager.cc:1905] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:14.227038 7791 ts_tablet_manager.cc:1918] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:14.227162 7791 log.cc:1196] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/fbde041396cc4d0382c373f10845a05b
I20260929 00:57:14.227511 7792 ts_tablet_manager.cc:1918] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:14.227524 7791 ts_tablet_manager.cc:1939] T fbde041396cc4d0382c373f10845a05b P 1f94bd11037f41fcbc42aa7e7c501a56: Deleting consensus metadata
I20260929 00:57:14.227619 7792 log.cc:1196] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/fbde041396cc4d0382c373f10845a05b
I20260929 00:57:14.228022 7792 ts_tablet_manager.cc:1939] T fbde041396cc4d0382c373f10845a05b P 7b2561af2bd74f91b039b18986c2ef01: Deleting consensus metadata
W20260929 00:57:14.229259 7623 connection.cc:441] server connection from 127.0.0.1:42982 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42982 (request call id 1) could send its response
W20260929 00:57:14.229813 7486 connection.cc:535] server connection from 127.0.0.1:60248 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 00:57:14.230974 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.219022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.0.98.61:38657)
0929 00:57:14.219144 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:14.219151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:14.219403 (+ 252us) server_negotiation.cc:407] Connection header received
0929 00:57:14.219625 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:14.219628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:14.219681 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:14.219766 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:14.221822 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.222649 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.223913 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.224245 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.224330 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.226541 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:14.226546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:14.226549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:14.226576 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:14.228925 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.229441 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.229445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.229447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.229542 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.229874 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.229879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.229881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.230123 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:14.230275 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:14.230663 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 00:57:14.230790 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:57:14.240613 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.220173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.0.98.60:41431)
0929 00:57:14.220445 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:14.220453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:14.231110 (+ 10657us) server_negotiation.cc:407] Connection header received
0929 00:57:14.231293 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:14.231297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:14.231351 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:14.231400 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:14.232235 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.232996 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.233984 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.234366 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.234492 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.236185 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:14.236189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:14.236192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:14.236216 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:14.237964 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.238459 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.238464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.238467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.238565 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.238832 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.238835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.238836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.239050 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:14.239646 (+ 596us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:14.240277 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 00:57:14.240399 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":169}
I20260929 00:57:14.241578 7793 tablet_replica.cc:330] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: stopping tablet replica
I20260929 00:57:14.242026 7793 raft_consensus.cc:2229] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:14.242491 7793 raft_consensus.cc:2258] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:14.244002 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.233144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.0.98.60:41431)
0929 00:57:14.233242 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:14.233246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:14.235300 (+ 2054us) server_negotiation.cc:407] Connection header received
0929 00:57:14.235480 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:14.235485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:14.235538 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:14.235596 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:14.236432 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.237261 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.238511 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.238838 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.238913 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.240610 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:14.240614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:14.240616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:14.240641 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:14.242171 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.242723 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.242726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.242728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.242807 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.243113 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.243117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.243118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.243306 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:14.243412 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:14.243752 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 00:57:14.243858 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 00:57:14.244516 7793 ts_tablet_manager.cc:1905] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:14.245409 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.234652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.0.98.61:38657)
0929 00:57:14.234794 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:14.234799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:14.234814 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:14.234850 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:14.234853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:14.234897 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:14.234980 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:14.236019 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.236881 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.238248 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.238628 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.238709 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.240496 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:14.240500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:14.240503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:14.240529 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:14.243106 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.243697 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.243701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.243702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.243793 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.244351 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.244355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.244356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.244595 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:14.244697 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:14.245124 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 00:57:14.245244 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:57:14.247398 7793 ts_tablet_manager.cc:1918] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:14.247486 7793 log.cc:1196] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/fbde041396cc4d0382c373f10845a05b
I20260929 00:57:14.247742 7793 ts_tablet_manager.cc:1939] T fbde041396cc4d0382c373f10845a05b P 126acb8caee2443caf765bd68a5bcf62: Deleting consensus metadata
W20260929 00:57:14.248883 7349 connection.cc:441] server connection from 127.0.0.1:53208 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53208 (request call id 1) could send its response
I20260929 00:57:15.207957 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.207401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:47875)
0929 00:57:15.207716 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.207829 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":92,"threads_started":1}
W20260929 00:57:15.208307 7600 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
I20260929 00:57:15.209772 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.209306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:59601)
0929 00:57:15.209587 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.209676 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
I20260929 00:57:15.210016 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.209435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:43051)
0929 00:57:15.209639 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.209707 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
W20260929 00:57:15.210346 7463 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
W20260929 00:57:15.210407 7736 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
I20260929 00:57:15.718166 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bfa9bcd8f5854b3ea41ad73fd185b311)
I20260929 00:57:15.718464 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bfa9bcd8f5854b3ea41ad73fd185b311)
I20260929 00:57:15.718468 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } }
I20260929 00:57:15.718633 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } }
I20260929 00:57:15.719520 7805 leader_election.cc:288] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915), 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60:41431)
I20260929 00:57:15.719552 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915), dc364728f1d540e099d9e2a51aae1e18 (127.0.98.61:38657)
I20260929 00:57:15.720074 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.719651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49156)
0929 00:57:15.719874 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.719964 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260929 00:57:15.720114 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.719655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49140)
0929 00:57:15.719947 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.720019 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
W20260929 00:57:15.720269 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
W20260929 00:57:15.720279 7268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
I20260929 00:57:15.721299 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.721002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49170)
0929 00:57:15.721139 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.721187 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 00:57:15.721312 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.720866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49158)
0929 00:57:15.721204 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.721255 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
W20260929 00:57:15.721413 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
W20260929 00:57:15.721429 7084 leader_election.cc:334] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
I20260929 00:57:15.725028 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.720465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.0.98.60:41431)
0929 00:57:15.720826 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:15.720834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:15.720847 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:57:15.720895 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:15.720898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:15.720953 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:15.721040 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:15.721288 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:15.722250 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:15.723780 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:15.724378 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:15.724499 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:15.724552 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:15.724785 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 00:57:15.724882 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":302,"thread_start_us":71,"threads_started":1}
I20260929 00:57:15.725028 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.720274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.0.98.61:38657)
0929 00:57:15.720683 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:15.720688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:15.720709 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:57:15.720867 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:15.720870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:15.720912 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:15.721032 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:15.721390 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:15.722250 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:15.723786 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:15.724378 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:15.724493 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:15.724552 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:15.724786 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 00:57:15.724882 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":349,"thread_start_us":56,"threads_started":1}
I20260929 00:57:15.725077 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.719958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41431 (local address 127.0.0.1:60594)
0929 00:57:15.720287 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.720304 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:15.720499 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:15.721063 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:15.721066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:15.721078 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:15.721172 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:15.721178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:15.722367 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:15.722370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:15.723651 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:15.723661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:15.724524 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:15.724529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:15.724637 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:15.724664 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:57:15.724765 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:57:15.724921 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":69,"thread_start_us":57,"threads_started":1}
I20260929 00:57:15.725077 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.719906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38657 (local address 127.0.0.1:49064)
0929 00:57:15.720652 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.720665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:15.720743 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:15.721153 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:15.721156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:15.721173 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:15.721278 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:15.721285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:15.722367 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:15.722370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:15.723651 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:15.723661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:15.724524 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:15.724529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:15.724638 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:15.724664 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 00:57:15.724778 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 00:57:15.724918 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":51,"thread_start_us":60,"threads_started":1}
I20260929 00:57:15.725646 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc364728f1d540e099d9e2a51aae1e18" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fb28318fdca4d3dae6657bfbded69cc" is_pre_election: true
I20260929 00:57:15.725731 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fb28318fdca4d3dae6657bfbded69cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc364728f1d540e099d9e2a51aae1e18" is_pre_election: true
I20260929 00:57:15.725883 7292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc364728f1d540e099d9e2a51aae1e18 in term 1.
I20260929 00:57:15.725924 7109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb28318fdca4d3dae6657bfbded69cc in term 1.
I20260929 00:57:15.726294 7270 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fb28318fdca4d3dae6657bfbded69cc, dc364728f1d540e099d9e2a51aae1e18; no voters: bfa9bcd8f5854b3ea41ad73fd185b311
I20260929 00:57:15.726292 7084 leader_election.cc:302] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fb28318fdca4d3dae6657bfbded69cc, dc364728f1d540e099d9e2a51aae1e18; no voters: bfa9bcd8f5854b3ea41ad73fd185b311
I20260929 00:57:15.726503 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 00:57:15.726573 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bfa9bcd8f5854b3ea41ad73fd185b311)
I20260929 00:57:15.726617 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:57:15.726692 7804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 00:57:15.726749 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader bfa9bcd8f5854b3ea41ad73fd185b311)
I20260929 00:57:15.726799 7804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:57:15.727845 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } }
I20260929 00:57:15.727845 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } }
I20260929 00:57:15.728230 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 2 election: Requested vote from peers bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915), dc364728f1d540e099d9e2a51aae1e18 (127.0.98.61:38657)
I20260929 00:57:15.728264 7805 leader_election.cc:288] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [CANDIDATE]: Term 2 election: Requested vote from peers bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915), 4fb28318fdca4d3dae6657bfbded69cc (127.0.98.60:41431)
I20260929 00:57:15.728576 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.728348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49190)
0929 00:57:15.728432 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.728504 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 00:57:15.728593 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fb28318fdca4d3dae6657bfbded69cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc364728f1d540e099d9e2a51aae1e18"
I20260929 00:57:15.728799 7109 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fb28318fdca4d3dae6657bfbded69cc in current term 2: Already voted for candidate dc364728f1d540e099d9e2a51aae1e18 in this term.
I20260929 00:57:15.728796 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.728314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49182)
0929 00:57:15.728462 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.728588 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 00:57:15.728765 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc364728f1d540e099d9e2a51aae1e18" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fb28318fdca4d3dae6657bfbded69cc"
I20260929 00:57:15.729104 7292 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dc364728f1d540e099d9e2a51aae1e18 in current term 2: Already voted for candidate 4fb28318fdca4d3dae6657bfbded69cc in this term.
I20260929 00:57:15.729579 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.729388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49210)
0929 00:57:15.729478 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.729523 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 00:57:15.729578 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:15.729371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49204)
0929 00:57:15.729473 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:15.729523 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 00:57:15.729717 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
I20260929 00:57:15.729822 7268 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fb28318fdca4d3dae6657bfbded69cc; no voters: bfa9bcd8f5854b3ea41ad73fd185b311, dc364728f1d540e099d9e2a51aae1e18
I20260929 00:57:15.729954 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260929 00:57:15.730089 7084 leader_election.cc:334] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
I20260929 00:57:15.730171 7084 leader_election.cc:302] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc364728f1d540e099d9e2a51aae1e18; no voters: 4fb28318fdca4d3dae6657bfbded69cc, bfa9bcd8f5854b3ea41ad73fd185b311
I20260929 00:57:15.730365 7805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 00:57:16.209585 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:16.209043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:43983)
0929 00:57:16.209380 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:16.209464 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
I20260929 00:57:16.211383 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:16.210926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:51981)
0929 00:57:16.211182 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:16.211251 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260929 00:57:16.211858 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:16.210953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:36545)
0929 00:57:16.211273 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:16.211752 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
I20260929 00:57:17.210960 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.210516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:44741)
0929 00:57:17.210805 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.210866 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260929 00:57:17.212527 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.212077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:52193)
0929 00:57:17.212338 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.212415 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":66,"threads_started":1}
I20260929 00:57:17.213593 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.212642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:59017)
0929 00:57:17.213420 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.213489 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":66,"threads_started":1}
I20260929 00:57:17.241446 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:17.241577 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } }
I20260929 00:57:17.241958 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915), dc364728f1d540e099d9e2a51aae1e18 (127.0.98.61:38657)
I20260929 00:57:17.242439 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fb28318fdca4d3dae6657bfbded69cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc364728f1d540e099d9e2a51aae1e18" is_pre_election: true
I20260929 00:57:17.242570 7109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb28318fdca4d3dae6657bfbded69cc in term 2.
I20260929 00:57:17.242663 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.242171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49364)
0929 00:57:17.242459 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.242516 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260929 00:57:17.242946 7270 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb28318fdca4d3dae6657bfbded69cc, dc364728f1d540e099d9e2a51aae1e18; no voters:
I20260929 00:57:17.243345 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 00:57:17.243396 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:17.243423 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:57:17.243498 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.243247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49378)
0929 00:57:17.243355 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.243426 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 00:57:17.243584 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
I20260929 00:57:17.244515 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } }
I20260929 00:57:17.244848 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 3 election: Requested vote from peers bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915), dc364728f1d540e099d9e2a51aae1e18 (127.0.98.61:38657)
I20260929 00:57:17.245253 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.245001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49384)
0929 00:57:17.245136 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.245196 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 00:57:17.245244 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fb28318fdca4d3dae6657bfbded69cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc364728f1d540e099d9e2a51aae1e18"
I20260929 00:57:17.245378 7109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:57:17.245895 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.245686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49390)
0929 00:57:17.245794 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.245843 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 00:57:17.246008 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
I20260929 00:57:17.246573 7109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb28318fdca4d3dae6657bfbded69cc in term 3.
I20260929 00:57:17.246860 7270 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fb28318fdca4d3dae6657bfbded69cc, dc364728f1d540e099d9e2a51aae1e18; no voters: bfa9bcd8f5854b3ea41ad73fd185b311
I20260929 00:57:17.247041 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 3 FOLLOWER]: Leader election won for term 3
I20260929 00:57:17.247247 7820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [term 3 LEADER]: Becoming Leader. State: Replica: 4fb28318fdca4d3dae6657bfbded69cc, State: Running, Role: LEADER
I20260929 00:57:17.247516 7820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } }
I20260929 00:57:17.248775 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb28318fdca4d3dae6657bfbded69cc. Latest consensus state: current_term: 3 leader_uuid: "4fb28318fdca4d3dae6657bfbded69cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:17.248895 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:17.249178 7825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:17.251385 7825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:17.251775 7825 catalog_manager.cc:1259] Loaded cluster ID: abd1102f78924fdb8fbf08382611a176
I20260929 00:57:17.251821 7825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:17.252442 7825 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:17.252835 7825 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Loaded TSK: 0
I20260929 00:57:17.253504 7825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:17.310081 7282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:17.313383 7282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:17.316571 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.316315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49402)
0929 00:57:17.316450 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.316503 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 00:57:17.316599 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [term 3 FOLLOWER]: Refusing update from remote peer 4fb28318fdca4d3dae6657bfbded69cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 00:57:17.317090 7820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 00:57:17.317323 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.317083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:49414)
0929 00:57:17.317194 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.317254 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 00:57:17.317444 7268 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc -> Peer bfa9bcd8f5854b3ea41ad73fd185b311 (127.0.98.62:45915): Couldn't send request to peer bfa9bcd8f5854b3ea41ad73fd185b311. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 00:57:17.318781 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb28318fdca4d3dae6657bfbded69cc. Latest consensus state: current_term: 3 leader_uuid: "4fb28318fdca4d3dae6657bfbded69cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:17.318935 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:17.320082 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4fb28318fdca4d3dae6657bfbded69cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:17.320194 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:17.320461 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4fb28318fdca4d3dae6657bfbded69cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bfa9bcd8f5854b3ea41ad73fd185b311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } peers { permanent_uuid: "dc364728f1d540e099d9e2a51aae1e18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38657 } attrs { promote: false } } peers { permanent_uuid: "4fb28318fdca4d3dae6657bfbded69cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41431 } attrs { promote: false } } }
I20260929 00:57:17.320652 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc364728f1d540e099d9e2a51aae1e18 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:17.331386 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.326330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.0.98.3:36963)
0929 00:57:17.327049 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.327054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.327070 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:57:17.327168 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.327171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.327242 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.327650 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:17.327771 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.328443 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.330269 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.330798 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.330928 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.330997 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.331164 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.331239 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":98,"threads_started":1}
I20260929 00:57:17.331382 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.326135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36963 (local address 127.0.0.1:42994)
0929 00:57:17.326311 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.326326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.326423 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.327373 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.327380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.327399 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:17.327514 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.327521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.328593 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.328597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.330083 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.330093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.330927 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.330931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.331027 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.331072 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.331149 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.331232 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":62}
I20260929 00:57:17.331815 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.326724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35887 (local address 127.0.0.1:60254)
0929 00:57:17.327216 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.327231 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.327328 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.327720 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.327723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.327739 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:17.327827 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.327832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.328875 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.328882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.330329 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.330338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.331396 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.331400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.331502 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.331533 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.331594 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.331676 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":26,"thread_start_us":86,"threads_started":1}
I20260929 00:57:17.332243 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.326817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.0.98.2:35887)
0929 00:57:17.327082 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.327088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.327416 (+ 328us) server_negotiation.cc:407] Connection header received
0929 00:57:17.327472 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.327474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.327523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.327609 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:17.327970 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.328745 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.330452 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.331187 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.331294 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.331988 (+ 694us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.332030 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.332104 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
I20260929 00:57:17.332268 7669 tablet_service.cc:1467] Processing CreateTablet for tablet 1e21eb596d9f4a77ae620abfd0053103 (DEFAULT_TABLE table=test-table [id=6b4c0abbc59247b9858db186a9bf0d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:17.332648 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e21eb596d9f4a77ae620abfd0053103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:17.332909 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.326535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40761 (local address 127.0.0.1:53216)
0929 00:57:17.326865 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.326880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.326972 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.328921 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.328924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.328943 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:17.329105 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.329112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.330114 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.330121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.331427 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.331436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.332469 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.332476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.332580 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.332599 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.332711 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.332779 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260929 00:57:17.332971 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.326865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.0.98.1:40761)
0929 00:57:17.328700 (+ 1835us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.328703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.328715 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:57:17.328758 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.328760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.328798 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.328914 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:17.329246 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.329960 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.331562 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.332220 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.332338 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.332405 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.332730 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.332833 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":89,"threads_started":1}
I20260929 00:57:17.333814 7396 tablet_service.cc:1467] Processing CreateTablet for tablet 1e21eb596d9f4a77ae620abfd0053103 (DEFAULT_TABLE table=test-table [id=6b4c0abbc59247b9858db186a9bf0d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:17.334113 7396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e21eb596d9f4a77ae620abfd0053103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:17.335312 7837 tablet_bootstrap.cc:492] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: Bootstrap starting.
I20260929 00:57:17.335811 7533 tablet_service.cc:1467] Processing CreateTablet for tablet 1e21eb596d9f4a77ae620abfd0053103 (DEFAULT_TABLE table=test-table [id=6b4c0abbc59247b9858db186a9bf0d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:17.336076 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e21eb596d9f4a77ae620abfd0053103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:17.336272 7837 tablet_bootstrap.cc:654] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:17.336598 7838 tablet_bootstrap.cc:492] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62: Bootstrap starting.
I20260929 00:57:17.337481 7838 tablet_bootstrap.cc:654] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:17.338660 7841 tablet_bootstrap.cc:492] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: Bootstrap starting.
I20260929 00:57:17.338896 7837 tablet_bootstrap.cc:492] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: No bootstrap required, opened a new log
I20260929 00:57:17.338989 7837 ts_tablet_manager.cc:1397] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 00:57:17.339425 7837 raft_consensus.cc:348] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.339579 7841 tablet_bootstrap.cc:654] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:17.339587 7837 raft_consensus.cc:374] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:17.339627 7837 raft_consensus.cc:729] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2561af2bd74f91b039b18986c2ef01, State: Initialized, Role: FOLLOWER
I20260929 00:57:17.339778 7837 consensus_queue.cc:260] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.340178 7837 ts_tablet_manager.cc:1428] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 00:57:17.340816 7838 tablet_bootstrap.cc:492] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62: No bootstrap required, opened a new log
I20260929 00:57:17.340904 7838 ts_tablet_manager.cc:1397] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 00:57:17.341173 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.340890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:46379)
0929 00:57:17.341045 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.341101 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 00:57:17.341327 7838 raft_consensus.cc:348] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.341428 7838 raft_consensus.cc:374] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:17.341468 7838 raft_consensus.cc:729] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126acb8caee2443caf765bd68a5bcf62, State: Initialized, Role: FOLLOWER
I20260929 00:57:17.341606 7838 consensus_queue.cc:260] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.341928 7838 ts_tablet_manager.cc:1428] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:17.342782 7735 heartbeater.cc:503] Master 127.0.98.60:41431 was elected leader, sending a full tablet report...
I20260929 00:57:17.343369 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.343074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:58101)
0929 00:57:17.343227 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.343295 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 00:57:17.344138 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.343910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:35637)
0929 00:57:17.344018 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.344067 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 00:57:17.344455 7462 heartbeater.cc:503] Master 127.0.98.60:41431 was elected leader, sending a full tablet report...
I20260929 00:57:17.344663 7841 tablet_bootstrap.cc:492] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: No bootstrap required, opened a new log
I20260929 00:57:17.344771 7841 ts_tablet_manager.cc:1397] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 00:57:17.345238 7841 raft_consensus.cc:348] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.345340 7841 raft_consensus.cc:374] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:17.345363 7841 raft_consensus.cc:729] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f94bd11037f41fcbc42aa7e7c501a56, State: Initialized, Role: FOLLOWER
I20260929 00:57:17.345525 7841 consensus_queue.cc:260] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.345887 7841 ts_tablet_manager.cc:1428] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:17.346508 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.346241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:41537)
0929 00:57:17.346349 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.346436 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 00:57:17.347208 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.346952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:35841)
0929 00:57:17.347090 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.347135 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 00:57:17.350592 7599 heartbeater.cc:503] Master 127.0.98.60:41431 was elected leader, sending a full tablet report...
I20260929 00:57:17.670090 7842 raft_consensus.cc:482] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:17.670264 7842 raft_consensus.cc:504] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.671520 7842 leader_election.cc:288] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761), 1f94bd11037f41fcbc42aa7e7c501a56 (127.0.98.2:35887)
I20260929 00:57:17.675384 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.671918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50837 (local address 127.0.98.2:35887)
0929 00:57:17.672042 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.672048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.672120 (+ 72us) server_negotiation.cc:407] Connection header received
0929 00:57:17.672255 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.672261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.672315 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.672412 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:17.672717 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.673304 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.674434 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.674832 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.674949 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.675014 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.675189 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.675265 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:57:17.675384 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.671838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35887 (local address 127.0.98.3:50837)
0929 00:57:17.671979 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.671995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.672122 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.672425 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.672429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.672443 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:17.672606 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.672619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.673418 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.673423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.674316 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.674325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.674958 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.674961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.675051 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.675073 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.675174 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.675245 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:57:17.675529 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.671929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42535 (local address 127.0.98.1:40761)
0929 00:57:17.672044 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.672049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.672403 (+ 354us) server_negotiation.cc:407] Connection header received
0929 00:57:17.672530 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.672533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.672575 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.672673 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:17.672952 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.673505 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.674579 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.674968 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.675083 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.675141 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.675331 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.675401 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 00:57:17.675575 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.671847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40761 (local address 127.0.98.3:42535)
0929 00:57:17.672309 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.672319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.672425 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.672688 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.672691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.672709 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:17.672833 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.672847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.673614 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.673620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.674463 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.674472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.675097 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.675109 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.675214 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.675230 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.675318 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.675382 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":183,"thread_start_us":125,"threads_started":1}
I20260929 00:57:17.676095 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e21eb596d9f4a77ae620abfd0053103" candidate_uuid: "7b2561af2bd74f91b039b18986c2ef01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126acb8caee2443caf765bd68a5bcf62" is_pre_election: true
I20260929 00:57:17.676139 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e21eb596d9f4a77ae620abfd0053103" candidate_uuid: "7b2561af2bd74f91b039b18986c2ef01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" is_pre_election: true
I20260929 00:57:17.676266 7416 raft_consensus.cc:2454] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b2561af2bd74f91b039b18986c2ef01 in term 0.
I20260929 00:57:17.676363 7553 raft_consensus.cc:2454] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b2561af2bd74f91b039b18986c2ef01 in term 0.
I20260929 00:57:17.676704 7625 leader_election.cc:302] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 126acb8caee2443caf765bd68a5bcf62, 7b2561af2bd74f91b039b18986c2ef01; no voters:
I20260929 00:57:17.676961 7842 raft_consensus.cc:2781] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:17.677033 7842 raft_consensus.cc:482] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:17.677088 7842 raft_consensus.cc:3037] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:17.678077 7842 raft_consensus.cc:504] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.678448 7842 leader_election.cc:288] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [CANDIDATE]: Term 1 election: Requested vote from peers 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761), 1f94bd11037f41fcbc42aa7e7c501a56 (127.0.98.2:35887)
I20260929 00:57:17.678849 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e21eb596d9f4a77ae620abfd0053103" candidate_uuid: "7b2561af2bd74f91b039b18986c2ef01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f94bd11037f41fcbc42aa7e7c501a56"
I20260929 00:57:17.678920 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e21eb596d9f4a77ae620abfd0053103" candidate_uuid: "7b2561af2bd74f91b039b18986c2ef01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126acb8caee2443caf765bd68a5bcf62"
I20260929 00:57:17.678978 7553 raft_consensus.cc:3037] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:17.679042 7416 raft_consensus.cc:3037] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:17.679872 7416 raft_consensus.cc:2454] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b2561af2bd74f91b039b18986c2ef01 in term 1.
I20260929 00:57:17.679962 7553 raft_consensus.cc:2454] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b2561af2bd74f91b039b18986c2ef01 in term 1.
I20260929 00:57:17.680153 7625 leader_election.cc:302] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 126acb8caee2443caf765bd68a5bcf62, 7b2561af2bd74f91b039b18986c2ef01; no voters:
I20260929 00:57:17.680341 7842 raft_consensus.cc:2781] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:17.680573 7842 raft_consensus.cc:686] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 1 LEADER]: Becoming Leader. State: Replica: 7b2561af2bd74f91b039b18986c2ef01, State: Running, Role: LEADER
I20260929 00:57:17.681031 7842 consensus_queue.cc:237] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } }
I20260929 00:57:17.681314 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.681038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:53201)
0929 00:57:17.681182 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.681242 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 00:57:17.683636 7283 catalog_manager.cc:5495] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b2561af2bd74f91b039b18986c2ef01 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7b2561af2bd74f91b039b18986c2ef01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2561af2bd74f91b039b18986c2ef01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:17.696241 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.692786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.98.3:36963)
0929 00:57:17.692897 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.692901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.692913 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:57:17.692976 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.692978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.693031 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.693116 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:17.693332 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.693819 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.694857 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.695061 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.695139 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.695389 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:17.695860 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:17.695950 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.696041 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.696087 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 00:57:17.699007 7416 raft_consensus.cc:1261] T 1e21eb596d9f4a77ae620abfd0053103 P 126acb8caee2443caf765bd68a5bcf62 [term 1 FOLLOWER]: Refusing update from remote peer 7b2561af2bd74f91b039b18986c2ef01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:17.699180 7553 raft_consensus.cc:1261] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 FOLLOWER]: Refusing update from remote peer 7b2561af2bd74f91b039b18986c2ef01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:17.699620 7847 consensus_queue.cc:1035] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "126acb8caee2443caf765bd68a5bcf62" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:17.699795 7842 consensus_queue.cc:1035] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f94bd11037f41fcbc42aa7e7c501a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:17.702446 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.701857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:33527)
0929 00:57:17.702305 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.702367 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260929 00:57:17.702699 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.702373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:43357)
0929 00:57:17.702581 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.702633 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 00:57:17.705780 7851 mvcc.cc:205] Tried to move back new op lower bound from 7334475520810057728 to 7334475520744271872. Current Snapshot: MvccSnapshot[applied={T|T < 7334475520810057728}]
I20260929 00:57:17.706359 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.705498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.1:47425)
0929 00:57:17.706217 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.706287 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624}
I20260929 00:57:17.706707 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.705940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:34953)
0929 00:57:17.706579 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.706635 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567}
I20260929 00:57:17.706712 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.706378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.2:36175)
0929 00:57:17.706580 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.706636 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 00:57:17.709959 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.709586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.98.3:54811)
0929 00:57:17.709804 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.709872 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260929 00:57:17.715454 7283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:17.715622 7283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:17.719376 7283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fb28318fdca4d3dae6657bfbded69cc: Sending DeleteTablet for 3 replicas of tablet 1e21eb596d9f4a77ae620abfd0053103
I20260929 00:57:17.719990 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e21eb596d9f4a77ae620abfd0053103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:17 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42994
I20260929 00:57:17.720129 7533 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e21eb596d9f4a77ae620abfd0053103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:17 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60254
I20260929 00:57:17.720376 7860 tablet_replica.cc:330] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: stopping tablet replica
I20260929 00:57:17.720494 7861 tablet_replica.cc:330] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: stopping tablet replica
I20260929 00:57:17.720552 7860 raft_consensus.cc:2229] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:17.720623 7861 raft_consensus.cc:2229] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:17.720803 7861 raft_consensus.cc:2258] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:17.720897 7860 raft_consensus.cc:2258] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:17.720944 7396 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e21eb596d9f4a77ae620abfd0053103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:17 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53216
I20260929 00:57:17.721623 7860 ts_tablet_manager.cc:1905] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:17.721623 7861 ts_tablet_manager.cc:1905] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:17.721948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 7243
I20260929 00:57:17.725626 7860 ts_tablet_manager.cc:1918] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:17.725626 7861 ts_tablet_manager.cc:1918] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:17.725713 7861 log.cc:1196] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/1e21eb596d9f4a77ae620abfd0053103
I20260929 00:57:17.725713 7860 log.cc:1196] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/1e21eb596d9f4a77ae620abfd0053103
I20260929 00:57:17.726006 7861 ts_tablet_manager.cc:1939] T 1e21eb596d9f4a77ae620abfd0053103 P 1f94bd11037f41fcbc42aa7e7c501a56: Deleting consensus metadata
I20260929 00:57:17.726006 7860 ts_tablet_manager.cc:1939] T 1e21eb596d9f4a77ae620abfd0053103 P 7b2561af2bd74f91b039b18986c2ef01: Deleting consensus metadata
I20260929 00:57:17.726842 7267 catalog_manager.cc:4841] TS 1f94bd11037f41fcbc42aa7e7c501a56 (127.0.98.2:35887): tablet 1e21eb596d9f4a77ae620abfd0053103 (table test-table [id=6b4c0abbc59247b9858db186a9bf0d45]) successfully deleted
I20260929 00:57:17.727025 7267 catalog_manager.cc:4841] TS 7b2561af2bd74f91b039b18986c2ef01 (127.0.98.3:36963): tablet 1e21eb596d9f4a77ae620abfd0053103 (table test-table [id=6b4c0abbc59247b9858db186a9bf0d45]) successfully deleted
I20260929 00:57:17.731449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 7470
I20260929 00:57:17.731676 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.731369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40761 (local address 127.0.0.1:53220)
0929 00:57:17.731511 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.731576 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 00:57:17.732620 7270 catalog_manager.cc:4584] TS 126acb8caee2443caf765bd68a5bcf62 (127.0.98.1:40761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e21eb596d9f4a77ae620abfd0053103: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40761: connect: Connection refused (error 111)
I20260929 00:57:17.738348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 7606
I20260929 00:57:17.743957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 6962
I20260929 00:57:17.749617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 7065
2026-09-29T00:57:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7501 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:17 dist-test-slave-30s6 krb5kdc[7865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:17 dist-test-slave-30s6 krb5kdc[7865](info): set up 2 sockets
Sep 29 00:57:17 dist-test-slave-30s6 krb5kdc[7865](info): commencing operation
krb5kdc: starting...
W20260929 00:57:18.822953 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:18 dist-test-slave-30s6 krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 00:57:18.941519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:19.051694 7878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:19.052026 7878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:19.052114 7878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:19.056154 7878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:19.056257 7878 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:57:19.056291 7878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:19.056332 7878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:19.056370 7878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:19.056420 7878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:19.061744 7878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36883
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:19.062873 7878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:19.063908 7878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:19.073324 7884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:19.073309 7886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:19.073374 7878 server_base.cc:1034] running on GCE node
W20260929 00:57:19.073309 7883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:19.074311 7878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:19.075469 7878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:19.076671 7878 hybrid_clock.cc:648] HybridClock initialized: now 1790643439076658 us; error 33 us; skew 500 ppm
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:19.079939 7878 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 00:57:19.081004 7878 webserver.cc:460] Webserver started at http://127.0.98.62:38135/ using document root <none> and password file <none>
I20260929 00:57:19.081632 7878 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:19.081737 7878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:19.081996 7878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:19.084422 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "325267e44ac446f49041080a48172c1c"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "1ea0e3596f0fc6d40201048a2d3047c7"
server_key_iv: "c751b784bf04ecddb185f00b9476af0b"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.085083 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "325267e44ac446f49041080a48172c1c"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "1ea0e3596f0fc6d40201048a2d3047c7"
server_key_iv: "c751b784bf04ecddb185f00b9476af0b"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.089313 7878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 00:57:19.092094 7893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:19.093366 7878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:19.093528 7878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "325267e44ac446f49041080a48172c1c"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "1ea0e3596f0fc6d40201048a2d3047c7"
server_key_iv: "c751b784bf04ecddb185f00b9476af0b"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.093653 7878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:19.109920 7878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:19.113727 7878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:19.113989 7878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:19.123121 7878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36883
I20260929 00:57:19.123138 7955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36883 every 8 connection(s)
I20260929 00:57:19.124212 7878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:19.127100 7956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:19.128105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 7878
I20260929 00:57:19.128253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:19.128621 392 external_mini_cluster.cc:1428] Setting key 348ac9734525ecfe282b2ea0071a6ded
I20260929 00:57:19.133073 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: Bootstrap starting.
I20260929 00:57:19.135416 7956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:19.136394 7956 log.cc:824] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:19.138607 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: No bootstrap required, opened a new log
I20260929 00:57:19.141911 7956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325267e44ac446f49041080a48172c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36883 } }
I20260929 00:57:19.142222 7956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:19.142315 7956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325267e44ac446f49041080a48172c1c, State: Initialized, Role: FOLLOWER
I20260929 00:57:19.142890 7956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325267e44ac446f49041080a48172c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36883 } }
I20260929 00:57:19.143067 7956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:19.143160 7956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:19.143261 7956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:19.143949 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.130229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.0.98.62:36883)
0929 00:57:19.130739 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.130764 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.130803 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:57:19.131476 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.131497 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.131849 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.132231 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.133045 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.134096 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.135021 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.135500 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.135680 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.138534 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:19.138539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:19.138553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:19.138580 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:19.140868 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.141329 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.141336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.141348 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.141460 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.141751 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.141756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.141758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.142169 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:19.142419 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.142736 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.143082 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":135,"threads_started":1}
I20260929 00:57:19.144320 7956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325267e44ac446f49041080a48172c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36883 } }
I20260929 00:57:19.144672 7956 leader_election.cc:302] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 325267e44ac446f49041080a48172c1c; no voters:
I20260929 00:57:19.145049 7956 leader_election.cc:288] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:19.145129 7961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:19.145318 7961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [term 1 LEADER]: Becoming Leader. State: Replica: 325267e44ac446f49041080a48172c1c, State: Running, Role: LEADER
I20260929 00:57:19.145674 7961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325267e44ac446f49041080a48172c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36883 } }
I20260929 00:57:19.146430 7956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:19.147727 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325267e44ac446f49041080a48172c1c. Latest consensus state: current_term: 1 leader_uuid: "325267e44ac446f49041080a48172c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325267e44ac446f49041080a48172c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36883 } } }
I20260929 00:57:19.147744 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "325267e44ac446f49041080a48172c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325267e44ac446f49041080a48172c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36883 } } }
I20260929 00:57:19.147907 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:19.148211 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:19.149093 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:19.152415 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:19.157675 7971 catalog_manager.cc:1347] Generated new cluster ID: daad26147ea24b6ab96661924966943f
I20260929 00:57:19.157747 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:19.180490 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:19.181429 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:19.190491 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: Generated new TSK 0
I20260929 00:57:19.191218 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 00:57:19.225075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36883
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:19.334662 7982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:19.334923 7982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:19.334968 7982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:19.338976 7982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:19.339037 7982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:19.339077 7982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:57:19.339109 7982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:19.344383 7982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:19.345364 7982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:19.346402 7982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:57:19.356168 7982 server_base.cc:1034] running on GCE node
W20260929 00:57:19.356197 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:19.356102 7990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:19.356350 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:19.356727 7982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:19.357417 7982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:19.358598 7982 hybrid_clock.cc:648] HybridClock initialized: now 1790643439358584 us; error 28 us; skew 500 ppm
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:19.361336 7982 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 00:57:19.362172 7982 webserver.cc:460] Webserver started at http://127.0.98.1:37201/ using document root <none> and password file <none>
I20260929 00:57:19.362907 7982 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:19.363001 7982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:19.363252 7982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:19.365232 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "601742aefcaa4d469c4587823fdc81ab"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "d9b46c908b91c3fbd539358bc4d5b949"
server_key_iv: "ade1fbfd73296315e4a76246386a4e83"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.365772 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "601742aefcaa4d469c4587823fdc81ab"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "d9b46c908b91c3fbd539358bc4d5b949"
server_key_iv: "ade1fbfd73296315e4a76246386a4e83"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.369321 7982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 00:57:19.371709 7997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:19.372856 7982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:57:19.372989 7982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "601742aefcaa4d469c4587823fdc81ab"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "d9b46c908b91c3fbd539358bc4d5b949"
server_key_iv: "ade1fbfd73296315e4a76246386a4e83"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.373097 7982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:19.392124 7982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:19.395156 7982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:19.395370 7982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:19.396874 7982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:19.396950 7982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:19.397014 7982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:19.397064 7982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:19.408306 7982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33221
I20260929 00:57:19.408383 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33221 every 8 connection(s)
I20260929 00:57:19.409416 7982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:19.411046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 7982
I20260929 00:57:19.411199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:19.411303 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.397699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51849 (local address 127.0.98.62:36883)
0929 00:57:19.397883 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.397890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.398830 (+ 940us) server_negotiation.cc:407] Connection header received
0929 00:57:19.399837 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.399841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.399897 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.399959 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.401833 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.402616 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.403915 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.404277 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.404399 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.407320 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:19.407323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:19.407325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:19.407351 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:19.409090 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.409592 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.409596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.409598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.409695 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.410107 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.410110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.410113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.410377 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:19.410490 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.410998 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.411123 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:19.411576 392 external_mini_cluster.cc:1428] Setting key f39e46baa1bbe9d1ff131fa1eeff9363
I20260929 00:57:19.412171 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.398170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36883 (local address 127.0.98.1:51849)
0929 00:57:19.398658 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.398708 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.399584 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.400445 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.400457 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.400957 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:19.401662 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.401684 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.402749 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.402752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.403776 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.403787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.404443 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.404447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.404628 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.405241 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:19.405261 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:19.407135 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:19.409204 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.409209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.409219 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.409469 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.409811 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.409815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.409817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.409979 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.410497 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:19.410505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:19.410757 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.410947 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.411260 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":125,"threads_started":1}
I20260929 00:57:19.413928 8113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36883
I20260929 00:57:19.414088 8113 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:19.414747 8113 heartbeater.cc:511] Master 127.0.98.62:36883 requested a full tablet report, sending...
I20260929 00:57:19.416463 7909 ts_manager.cc:194] Registered new tserver with Master: 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221)
I20260929 00:57:19.417593 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51849
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:57:19.423472 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.416255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57577 (local address 127.0.98.62:36883)
0929 00:57:19.416365 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.416368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.416579 (+ 211us) server_negotiation.cc:407] Connection header received
0929 00:57:19.416735 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.416738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.416780 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.416873 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.417300 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.417874 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.418848 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.419082 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.419196 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.420368 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:19.420372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:19.420375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:19.420397 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:19.421822 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.422287 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.422292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.422294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.422372 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.422646 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.422650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.422652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.422826 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:19.422934 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.423186 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.423298 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 00:57:19.423483 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.416151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36883 (local address 127.0.98.1:57577)
0929 00:57:19.416492 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.416504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.416633 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.416887 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.416890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.417093 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:19.417188 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.417194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.417994 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.417999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.418729 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.418738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.419206 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.419209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.419337 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.419877 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:19.419889 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:19.420223 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:19.421954 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.421957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.421960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.422159 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.422465 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.422469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.422471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.422556 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.422923 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:19.422926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:19.423021 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.423127 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.423267 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":5,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 00:57:19.441331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36883
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:19.556227 8122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:19.556505 8122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:19.556602 8122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:19.560778 8122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:19.560844 8122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:19.560884 8122 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:57:19.560917 8122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:19.566351 8122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:19.567449 8122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:19.568428 8122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:19.578253 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:19.578423 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:19.578475 8122 server_base.cc:1034] running on GCE node
W20260929 00:57:19.578281 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:19.578964 8122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:19.579648 8122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:19.580838 8122 hybrid_clock.cc:648] HybridClock initialized: now 1790643439580814 us; error 37 us; skew 500 ppm
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:19.583822 8122 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 00:57:19.584712 8122 webserver.cc:460] Webserver started at http://127.0.98.2:41403/ using document root <none> and password file <none>
I20260929 00:57:19.585278 8122 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:19.585369 8122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:19.585599 8122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:19.587719 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "ecbf3738948e4892b6c53eb9a6d12fe4"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "ada0a7fa7ff9a6bb70d6da2490a67758"
server_key_iv: "238fe3f1957e35213ef7af7d0ee57d5a"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.588294 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ecbf3738948e4892b6c53eb9a6d12fe4"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "ada0a7fa7ff9a6bb70d6da2490a67758"
server_key_iv: "238fe3f1957e35213ef7af7d0ee57d5a"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.592120 8122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:19.594761 8137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:19.596020 8122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:19.596189 8122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "ecbf3738948e4892b6c53eb9a6d12fe4"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "ada0a7fa7ff9a6bb70d6da2490a67758"
server_key_iv: "238fe3f1957e35213ef7af7d0ee57d5a"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.596341 8122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:19.611801 8122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:19.615600 8122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:19.615768 8122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:19.617226 8122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:19.617309 8122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:19.617379 8122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:19.617436 8122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:19.627944 8122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38443
I20260929 00:57:19.628005 8252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38443 every 8 connection(s)
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:19.628860 8122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:19.632503 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.618119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42719 (local address 127.0.98.62:36883)
0929 00:57:19.618292 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.618297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.619480 (+ 1183us) server_negotiation.cc:407] Connection header received
0929 00:57:19.620817 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.620821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.620876 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.620987 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.622564 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.623423 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.625259 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.625470 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.625603 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.628716 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:19.628719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:19.628722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:19.628751 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:19.630555 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.631016 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.631019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.631021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.631103 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.631523 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.631525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.631527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.631711 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:19.631788 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.632275 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.632381 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 00:57:19.633261 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.618611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36883 (local address 127.0.98.2:42719)
0929 00:57:19.619323 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.619374 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.620537 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.621131 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.621141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.621650 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:19.622395 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.622416 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.623567 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.623575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.625115 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.625129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.625688 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.625695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.625902 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.626667 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:19.626706 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:19.628541 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:19.630655 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.630660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.630670 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.630912 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.631193 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.631210 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.631214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.631402 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.631802 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:19.631807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:19.632026 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.632173 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.632417 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":184,"threads_started":1}
I20260929 00:57:19.634575 8253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36883
I20260929 00:57:19.634739 8253 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:19.635236 8253 heartbeater.cc:511] Master 127.0.98.62:36883 requested a full tablet report, sending...
I20260929 00:57:19.636298 7910 ts_manager.cc:194] Registered new tserver with Master: ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2:38443)
I20260929 00:57:19.637087 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42719
I20260929 00:57:19.638002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 8122
I20260929 00:57:19.638136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:19.638505 392 external_mini_cluster.cc:1428] Setting key 878a8dd055d38c915afcf00eba8c5d72
I20260929 00:57:19.644213 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.636690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36883 (local address 127.0.98.2:34321)
0929 00:57:19.636979 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.636992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.637097 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.637396 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.637398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.637630 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:19.637756 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.637766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.638609 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.638612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.639380 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.639389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.639830 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.639834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.639921 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.640459 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:19.640474 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:19.640917 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:19.642647 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.642651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.642654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.642860 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.643181 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.643184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.643187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.643275 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.643793 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:19.643795 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:19.643873 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.643961 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.644072 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":88,"threads_started":1}
I20260929 00:57:19.644410 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.636773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34321 (local address 127.0.98.62:36883)
0929 00:57:19.636881 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.636885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.637085 (+ 200us) server_negotiation.cc:407] Connection header received
0929 00:57:19.637234 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.637237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.637285 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.637365 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.637857 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.638492 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.639497 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.639720 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.639802 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.641063 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:19.641066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:19.641069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:19.641091 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:19.642511 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.642993 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.642997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.642999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.643082 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.643366 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.643372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.643377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.643687 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:19.643796 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.644043 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.644177 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 00:57:19.666891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36883
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:19.777408 8262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:19.777716 8262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:19.777767 8262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:19.781726 8262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:19.781826 8262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:19.781876 8262 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:57:19.781947 8262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:19.787112 8262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:19.788091 8262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:19.789081 8262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:57:19.798734 8262 server_base.cc:1034] running on GCE node
W20260929 00:57:19.798718 8268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:19.798713 8270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:19.798902 8267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:19.799379 8262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:19.800035 8262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:19.801216 8262 hybrid_clock.cc:648] HybridClock initialized: now 1790643439801175 us; error 54 us; skew 500 ppm
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:19.804066 8262 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 00:57:19.804984 8262 webserver.cc:460] Webserver started at http://127.0.98.3:45391/ using document root <none> and password file <none>
I20260929 00:57:19.805577 8262 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:19.805693 8262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:19.805922 8262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:19.808120 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "7dd45b0e0563c3f8fc066579dec24156"
server_key_iv: "e8327b2dc681a03e3c4027b56527289e"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.808704 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "7dd45b0e0563c3f8fc066579dec24156"
server_key_iv: "e8327b2dc681a03e3c4027b56527289e"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.812559 8262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 00:57:19.815315 8277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:19.816623 8262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:19.816774 8262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0"
format_stamp: "Formatted at 2026-09-29 00:57:19 on dist-test-slave-30s6"
server_key: "7dd45b0e0563c3f8fc066579dec24156"
server_key_iv: "e8327b2dc681a03e3c4027b56527289e"
server_key_version: "encryptionkey@0"
I20260929 00:57:19.816891 8262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:19.827054 8262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:19.829932 8262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:19.830132 8262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:19.831671 8262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:19.831749 8262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:19.831811 8262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:19.831873 8262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:19.843252 8262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40057
I20260929 00:57:19.843333 8392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40057 every 8 connection(s)
I20260929 00:57:19.844179 8262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:19.847069 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.832343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55105 (local address 127.0.98.62:36883)
0929 00:57:19.832483 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.832488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.833459 (+ 971us) server_negotiation.cc:407] Connection header received
0929 00:57:19.834780 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.834784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.834839 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.834925 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.836781 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.837387 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.839266 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.839586 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.839714 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.843031 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:19.843038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:19.843041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:19.843073 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:19.844707 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.845419 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.845422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.845424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.845513 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.845858 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.845862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.845863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.846135 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:19.846281 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.846787 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.846902 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:57:19.847986 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.832779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36883 (local address 127.0.98.3:55105)
0929 00:57:19.833271 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.833330 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.834513 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.835290 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.835302 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.835852 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:19.836597 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.836621 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.837497 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.837501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.839128 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.839142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.839762 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.839766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.839948 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.841056 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:19.841072 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:19.842856 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:19.844969 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.844975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.844988 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.845320 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.845612 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.845615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.845617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.845757 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.846278 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:19.846286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:19.846529 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.846733 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.847040 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260929 00:57:19.849125 8393 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36883
I20260929 00:57:19.849269 8393 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:19.849822 8393 heartbeater.cc:511] Master 127.0.98.62:36883 requested a full tablet report, sending...
I20260929 00:57:19.850862 7909 ts_manager.cc:194] Registered new tserver with Master: 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:19.851598 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55105
I20260929 00:57:19.853226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 8262
I20260929 00:57:19.853367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:19.853700 392 external_mini_cluster.cc:1428] Setting key 57fe71242f49e9d2d62c4f53f4e86b7c
I20260929 00:57:19.856035 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:57:19.858891 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.851500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58507 (local address 127.0.98.62:36883)
0929 00:57:19.851617 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.851620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.851876 (+ 256us) server_negotiation.cc:407] Connection header received
0929 00:57:19.851992 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.851995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.852038 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.852124 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.852605 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.853164 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.854174 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.854522 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.854639 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.855633 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:19.855637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:19.855640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:19.855663 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:19.857264 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.857661 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.857667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.857672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.857761 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.858024 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.858028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.858030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.858262 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:19.858368 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.858632 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.858744 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:57:19.858901 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.851413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36883 (local address 127.0.98.3:58507)
0929 00:57:19.851779 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.851793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.851900 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.852178 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.852181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.852388 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:19.852499 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.852506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.853303 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.853307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.854049 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.854061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.854645 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.854649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.854716 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.855191 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:19.855203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:19.855514 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:19.857378 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.857381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.857383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.857567 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.857849 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:19.857852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:19.857853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:19.857924 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:19.858369 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:19.858372 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:19.858466 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.858568 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.858708 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:57:19.879355 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:19.873558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.0.98.62:36883)
0929 00:57:19.873686 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.873690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.873702 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:57:19.873741 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.873743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.873780 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.873850 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.874565 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.875076 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.876036 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.876238 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.876314 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.879248 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58090: BlockingRecv error: recv got EOF from 127.0.0.1:58090 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 00:57:19 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:19.888347 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.880722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.0.98.62:36883)
0929 00:57:19.880847 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.880851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.880966 (+ 115us) server_negotiation.cc:407] Connection header received
0929 00:57:19.881109 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.881112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.881158 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.881242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:19.881808 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.882409 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.883187 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.883399 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.883501 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.885586 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:19.885589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:19.885591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:19.885615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:19.887053 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.887410 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.887412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.887414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.887482 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:19.887709 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:19.887711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:19.887712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:19.887865 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:19.887962 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.888153 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.888235 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:57:19.891402 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:19.894428 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:19.908746 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.902804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.0.98.2:38443)
0929 00:57:19.903604 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.903615 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.903639 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:57:19.903718 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.903726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.903936 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.904105 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:19.904433 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.905636 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.907541 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.908077 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.908211 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.908384 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.908494 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.908586 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":100,"threads_started":1}
I20260929 00:57:19.908735 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.902696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38443 (local address 127.0.0.1:36562)
0929 00:57:19.903131 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.903190 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.903316 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.904128 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.904136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.904162 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:19.904305 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.904311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.905768 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.905774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.907413 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.907423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.908212 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.908215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.908317 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.908352 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.908465 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.908586 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":230,"thread_start_us":75,"threads_started":1}
I20260929 00:57:19.909006 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.903372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40057 (local address 127.0.0.1:43614)
0929 00:57:19.904330 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.904341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.904449 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.904931 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.904933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.904951 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:19.905031 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.905036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.906316 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.906319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.907229 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.907238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.908496 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.908499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.908653 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.908693 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.908765 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.908865 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260929 00:57:19.909085 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.903486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.0.98.1:33221)
0929 00:57:19.903867 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.903881 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.904206 (+ 325us) server_negotiation.cc:407] Connection header received
0929 00:57:19.904281 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.904286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.904463 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.904624 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:19.904843 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.905680 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.907477 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.908192 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.908282 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.908433 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.908854 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.908946 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":83,"threads_started":1}
I20260929 00:57:19.909440 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.902774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33221 (local address 127.0.0.1:42110)
0929 00:57:19.903999 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:19.904010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:19.904115 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:19.904595 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:19.904598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:19.904615 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:19.904724 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.904733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.905822 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.905825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.907350 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:19.907360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.908496 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.908499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:19.908666 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.908703 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 00:57:19.908765 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 00:57:19.909319 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":652,"thread_start_us":113,"threads_started":1}
I20260929 00:57:19.909636 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:19.903786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.0.98.3:40057)
0929 00:57:19.904164 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:19.904175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:19.904549 (+ 374us) server_negotiation.cc:407] Connection header received
0929 00:57:19.904625 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:19.904630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:19.904803 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:19.904944 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:19.905168 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.906169 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.907355 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:19.908172 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:19.908282 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:19.908433 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:19.909376 (+ 943us) server_negotiation.cc:299] Negotiation successful
0929 00:57:19.909475 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":97,"threads_started":1}
I20260929 00:57:19.911926 8327 tablet_service.cc:1467] Processing CreateTablet for tablet c0718ba314a4471a912987645624d86d (DEFAULT_TABLE table=test-table [id=070b537adea7439eafe34c58040fbc0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:19.911931 8186 tablet_service.cc:1467] Processing CreateTablet for tablet c0718ba314a4471a912987645624d86d (DEFAULT_TABLE table=test-table [id=070b537adea7439eafe34c58040fbc0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:19.912796 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0718ba314a4471a912987645624d86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:19.912834 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0718ba314a4471a912987645624d86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:19.915318 8047 tablet_service.cc:1467] Processing CreateTablet for tablet c0718ba314a4471a912987645624d86d (DEFAULT_TABLE table=test-table [id=070b537adea7439eafe34c58040fbc0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:19.916395 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0718ba314a4471a912987645624d86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:19.919478 8421 tablet_bootstrap.cc:492] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: Bootstrap starting.
I20260929 00:57:19.921553 8421 tablet_bootstrap.cc:654] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:19.922341 8421 log.cc:824] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:19.922961 8422 tablet_bootstrap.cc:492] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Bootstrap starting.
I20260929 00:57:19.924417 8421 tablet_bootstrap.cc:492] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: No bootstrap required, opened a new log
I20260929 00:57:19.924615 8421 ts_tablet_manager.cc:1397] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:57:19.924722 8424 tablet_bootstrap.cc:492] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: Bootstrap starting.
I20260929 00:57:19.925124 8422 tablet_bootstrap.cc:654] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:19.925801 8422 log.cc:824] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:19.926733 8424 tablet_bootstrap.cc:654] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:19.927541 8422 tablet_bootstrap.cc:492] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: No bootstrap required, opened a new log
I20260929 00:57:19.927749 8422 ts_tablet_manager.cc:1397] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 00:57:19.927994 8424 log.cc:824] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:19.928944 8421 raft_consensus.cc:348] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:19.929215 8421 raft_consensus.cc:374] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:19.929301 8421 raft_consensus.cc:729] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Initialized, Role: FOLLOWER
I20260929 00:57:19.929759 8421 consensus_queue.cc:260] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:19.929903 8424 tablet_bootstrap.cc:492] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: No bootstrap required, opened a new log
I20260929 00:57:19.930109 8424 ts_tablet_manager.cc:1397] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:57:19.930465 8253 heartbeater.cc:503] Master 127.0.98.62:36883 was elected leader, sending a full tablet report...
I20260929 00:57:19.930804 8421 ts_tablet_manager.cc:1428] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:57:19.932186 8422 raft_consensus.cc:348] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:19.932412 8422 raft_consensus.cc:374] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:19.932484 8422 raft_consensus.cc:729] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e09e2a7a38240b18c60e6dc81f7a2c0, State: Initialized, Role: FOLLOWER
I20260929 00:57:19.932909 8422 consensus_queue.cc:260] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:19.933526 8393 heartbeater.cc:503] Master 127.0.98.62:36883 was elected leader, sending a full tablet report...
I20260929 00:57:19.933892 8422 ts_tablet_manager.cc:1428] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:57:19.933956 8424 raft_consensus.cc:348] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:19.934273 8424 raft_consensus.cc:374] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:19.934330 8424 raft_consensus.cc:729] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601742aefcaa4d469c4587823fdc81ab, State: Initialized, Role: FOLLOWER
I20260929 00:57:19.934787 8424 consensus_queue.cc:260] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:19.935417 8113 heartbeater.cc:503] Master 127.0.98.62:36883 was elected leader, sending a full tablet report...
I20260929 00:57:19.935797 8424 ts_tablet_manager.cc:1428] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:57:20.046434 8427 raft_consensus.cc:482] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:20.046741 8427 raft_consensus.cc:504] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.048061 8427 leader_election.cc:288] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.053022 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.048507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36287 (local address 127.0.98.1:33221)
0929 00:57:20.048646 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.048653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.048700 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:57:20.048931 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.048934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.048989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.049059 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:20.049466 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.050435 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.051935 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.052448 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.052572 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.052638 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.052829 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.052897 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:57:20.053040 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.048450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33221 (local address 127.0.98.2:36287)
0929 00:57:20.048612 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.048629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.048759 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.049180 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.049184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.049202 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:20.049332 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.049340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.050559 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.050562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.051788 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.051799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.052572 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.052576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.052691 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.052718 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.052810 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.052889 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 00:57:20.053177 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.048930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52667 (local address 127.0.98.3:40057)
0929 00:57:20.049075 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.049082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.049308 (+ 226us) server_negotiation.cc:407] Connection header received
0929 00:57:20.049434 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.049438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.049497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.049604 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:20.049837 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.050659 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.051852 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.052480 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.052589 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.052660 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.052959 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.053031 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:57:20.053426 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.048767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40057 (local address 127.0.98.2:52667)
0929 00:57:20.049214 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.049227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.049318 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.049615 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.049618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.049633 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:20.049724 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.049728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.050782 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.050788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.051708 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.051718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.052720 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.052724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.052821 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.052840 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.052939 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.053013 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":163,"threads_started":1}
I20260929 00:57:20.054077 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0718ba314a4471a912987645624d86d" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" is_pre_election: true
I20260929 00:57:20.054087 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0718ba314a4471a912987645624d86d" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601742aefcaa4d469c4587823fdc81ab" is_pre_election: true
I20260929 00:57:20.054443 8051 raft_consensus.cc:2454] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 0.
I20260929 00:57:20.054445 8347 raft_consensus.cc:2454] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 0.
I20260929 00:57:20.055023 8139 leader_election.cc:302] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e09e2a7a38240b18c60e6dc81f7a2c0, ecbf3738948e4892b6c53eb9a6d12fe4; no voters:
I20260929 00:57:20.055315 8427 raft_consensus.cc:2781] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:20.055393 8427 raft_consensus.cc:482] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:20.055423 8427 raft_consensus.cc:3037] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.056463 8427 raft_consensus.cc:504] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.056809 8427 leader_election.cc:288] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 election: Requested vote from peers 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.057399 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0718ba314a4471a912987645624d86d" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601742aefcaa4d469c4587823fdc81ab"
I20260929 00:57:20.057507 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0718ba314a4471a912987645624d86d" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0"
I20260929 00:57:20.057523 8051 raft_consensus.cc:3037] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.057678 8347 raft_consensus.cc:3037] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.058732 8051 raft_consensus.cc:2454] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 1.
I20260929 00:57:20.058888 8347 raft_consensus.cc:2454] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 1.
I20260929 00:57:20.059120 8139 leader_election.cc:302] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601742aefcaa4d469c4587823fdc81ab, ecbf3738948e4892b6c53eb9a6d12fe4; no voters:
I20260929 00:57:20.059310 8427 raft_consensus.cc:2781] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:20.059549 8427 raft_consensus.cc:686] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 LEADER]: Becoming Leader. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Running, Role: LEADER
I20260929 00:57:20.059902 8427 consensus_queue.cc:237] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.063033 7909 catalog_manager.cc:5495] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } health_report { overall_health: UNKNOWN } } }
W20260929 00:57:20.095491 8394 tablet.cc:2369] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:20.098634 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.094932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.0.98.2:38443)
0929 00:57:20.095077 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.095080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.095095 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:20.095171 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.095173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.095232 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.095328 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:20.095668 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.096190 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.097117 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.097347 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.097453 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.097709 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.098211 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.098299 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.098430 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.098478 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:57:20.104686 8347 raft_consensus.cc:1261] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Refusing update from remote peer ecbf3738948e4892b6c53eb9a6d12fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.104686 8051 raft_consensus.cc:1261] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Refusing update from remote peer ecbf3738948e4892b6c53eb9a6d12fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.105450 8427 consensus_queue.cc:1035] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.105724 8431 consensus_queue.cc:1035] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.114696 8435 mvcc.cc:205] Tried to move back new op lower bound from 7334475530661216256 to 7334475530488229888. Current Snapshot: MvccSnapshot[applied={T|T < 7334475530661216256}]
I20260929 00:57:20.122385 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:20.122575 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:20.125187 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: Sending DeleteTablet for 3 replicas of tablet c0718ba314a4471a912987645624d86d
I20260929 00:57:20.125942 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet c0718ba314a4471a912987645624d86d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42110
I20260929 00:57:20.125949 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet c0718ba314a4471a912987645624d86d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36562
I20260929 00:57:20.126500 8445 tablet_replica.cc:330] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: stopping tablet replica
I20260929 00:57:20.126892 8445 raft_consensus.cc:2229] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:20.126996 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet c0718ba314a4471a912987645624d86d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260929 00:57:20.127331 8445 raft_consensus.cc:2258] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.128432 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:20.128827 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:20.129035 8445 ts_tablet_manager.cc:1905] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.131819 8445 ts_tablet_manager.cc:1918] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:20.131997 8445 log.cc:1196] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/c0718ba314a4471a912987645624d86d
I20260929 00:57:20.132392 8445 ts_tablet_manager.cc:1939] T c0718ba314a4471a912987645624d86d P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting consensus metadata
I20260929 00:57:20.133646 7897 catalog_manager.cc:4841] TS ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2:38443): tablet c0718ba314a4471a912987645624d86d (table test-table [id=070b537adea7439eafe34c58040fbc0b]) successfully deleted
I20260929 00:57:20.134673 8446 tablet_replica.cc:330] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: stopping tablet replica
I20260929 00:57:20.134996 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 15612635644e498b809fd024bee15aa9 (DEFAULT_TABLE table=test-table [id=9093d77835f64264a8c9756a678544ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.135089 8446 raft_consensus.cc:2229] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:20.135247 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15612635644e498b809fd024bee15aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.135409 8446 raft_consensus.cc:2258] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.135782 8444 tablet_replica.cc:330] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: stopping tablet replica
I20260929 00:57:20.136065 8444 raft_consensus.cc:2229] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:20.136323 8444 raft_consensus.cc:2258] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.137478 8421 tablet_bootstrap.cc:492] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: Bootstrap starting.
I20260929 00:57:20.137940 8446 ts_tablet_manager.cc:1905] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.138460 8421 tablet_bootstrap.cc:654] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.139922 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 15612635644e498b809fd024bee15aa9 (DEFAULT_TABLE table=test-table [id=9093d77835f64264a8c9756a678544ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.139923 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 15612635644e498b809fd024bee15aa9 (DEFAULT_TABLE table=test-table [id=9093d77835f64264a8c9756a678544ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.140205 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15612635644e498b809fd024bee15aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.140242 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15612635644e498b809fd024bee15aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.140548 8446 ts_tablet_manager.cc:1918] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:20.140632 8446 log.cc:1196] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/c0718ba314a4471a912987645624d86d
I20260929 00:57:20.141219 8446 ts_tablet_manager.cc:1939] T c0718ba314a4471a912987645624d86d P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting consensus metadata
I20260929 00:57:20.142691 8422 tablet_bootstrap.cc:492] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Bootstrap starting.
I20260929 00:57:20.143261 8424 tablet_bootstrap.cc:492] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: Bootstrap starting.
I20260929 00:57:20.143625 8422 tablet_bootstrap.cc:654] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.143564 7895 catalog_manager.cc:4841] TS 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057): tablet c0718ba314a4471a912987645624d86d (table test-table [id=070b537adea7439eafe34c58040fbc0b]) successfully deleted
I20260929 00:57:20.144214 8424 tablet_bootstrap.cc:654] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.145303 8444 ts_tablet_manager.cc:1905] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.145699 8421 tablet_bootstrap.cc:492] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: No bootstrap required, opened a new log
I20260929 00:57:20.145800 8421 ts_tablet_manager.cc:1397] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 00:57:20.145905 8422 tablet_bootstrap.cc:492] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: No bootstrap required, opened a new log
I20260929 00:57:20.145991 8422 ts_tablet_manager.cc:1397] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 00:57:20.146342 8421 raft_consensus.cc:348] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.146478 8421 raft_consensus.cc:374] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.146436 8422 raft_consensus.cc:348] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.146519 8421 raft_consensus.cc:729] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.146566 8422 raft_consensus.cc:374] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.146610 8422 raft_consensus.cc:729] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e09e2a7a38240b18c60e6dc81f7a2c0, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.146692 8421 consensus_queue.cc:260] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.146741 8422 consensus_queue.cc:260] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.146981 8422 ts_tablet_manager.cc:1428] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:20.147006 8421 ts_tablet_manager.cc:1428] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 00:57:20.148531 8444 ts_tablet_manager.cc:1918] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:20.148661 8444 log.cc:1196] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/c0718ba314a4471a912987645624d86d
I20260929 00:57:20.149024 8444 ts_tablet_manager.cc:1939] T c0718ba314a4471a912987645624d86d P 601742aefcaa4d469c4587823fdc81ab: Deleting consensus metadata
I20260929 00:57:20.150616 7895 catalog_manager.cc:4841] TS 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221): tablet c0718ba314a4471a912987645624d86d (table test-table [id=070b537adea7439eafe34c58040fbc0b]) successfully deleted
I20260929 00:57:20.151728 8424 tablet_bootstrap.cc:492] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: No bootstrap required, opened a new log
I20260929 00:57:20.151808 8424 ts_tablet_manager.cc:1397] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260929 00:57:20.152426 8424 raft_consensus.cc:348] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.152530 8424 raft_consensus.cc:374] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.152554 8424 raft_consensus.cc:729] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601742aefcaa4d469c4587823fdc81ab, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.152717 8424 consensus_queue.cc:260] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.153005 8424 ts_tablet_manager.cc:1428] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260929 00:57:20.160874 8114 tablet.cc:2369] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:20.242484 8429 raft_consensus.cc:482] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:20.242658 8429 raft_consensus.cc:504] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.243966 8429 leader_election.cc:288] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2:38443), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.249188 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.244259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36707 (local address 127.0.98.2:38443)
0929 00:57:20.244385 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.244390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.244515 (+ 125us) server_negotiation.cc:407] Connection header received
0929 00:57:20.244740 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.244746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.244797 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.244849 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:20.245406 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.246228 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.247848 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.248511 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.248624 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.248713 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.248953 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.249033 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:57:20.249285 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.244642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52523 (local address 127.0.98.3:40057)
0929 00:57:20.244767 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.244772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.244789 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:57:20.244851 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.244854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.244900 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.244984 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:20.246087 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.246880 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.248236 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.248735 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.248862 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.248940 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.249087 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.249160 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:57:20.249316 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.244240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40057 (local address 127.0.98.1:52523)
0929 00:57:20.244692 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.244707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.244777 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.245085 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.245088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.245119 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:20.245272 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.245279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.247030 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.247034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.248127 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.248134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.248845 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.248849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.248956 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.248985 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.249073 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.249142 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":24,"spinlock_wait_cycles":1280,"thread_start_us":100,"threads_started":1}
I20260929 00:57:20.249816 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15612635644e498b809fd024bee15aa9" candidate_uuid: "601742aefcaa4d469c4587823fdc81ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" is_pre_election: true
I20260929 00:57:20.249981 8347 raft_consensus.cc:2454] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601742aefcaa4d469c4587823fdc81ab in term 0.
I20260929 00:57:20.250010 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.244151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38443 (local address 127.0.98.1:36707)
0929 00:57:20.244437 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.244453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.244598 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.244996 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.244999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.245013 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:20.245272 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.245279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.246361 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.246365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.247716 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.247726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.248674 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.248678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.248789 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.248821 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.249795 (+ 974us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.249857 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":25,"spinlock_wait_cycles":4224,"thread_start_us":74,"threads_started":1}
I20260929 00:57:20.250494 7999 leader_election.cc:302] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e09e2a7a38240b18c60e6dc81f7a2c0, 601742aefcaa4d469c4587823fdc81ab; no voters:
I20260929 00:57:20.250550 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15612635644e498b809fd024bee15aa9" candidate_uuid: "601742aefcaa4d469c4587823fdc81ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" is_pre_election: true
I20260929 00:57:20.250795 8207 raft_consensus.cc:2454] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601742aefcaa4d469c4587823fdc81ab in term 0.
I20260929 00:57:20.251087 8429 raft_consensus.cc:2781] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:20.251153 8429 raft_consensus.cc:482] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:20.251204 8429 raft_consensus.cc:3037] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.252171 8429 raft_consensus.cc:504] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.252540 8429 leader_election.cc:288] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [CANDIDATE]: Term 1 election: Requested vote from peers ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2:38443), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.252827 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15612635644e498b809fd024bee15aa9" candidate_uuid: "601742aefcaa4d469c4587823fdc81ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4"
I20260929 00:57:20.252871 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15612635644e498b809fd024bee15aa9" candidate_uuid: "601742aefcaa4d469c4587823fdc81ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0"
I20260929 00:57:20.252948 8207 raft_consensus.cc:3037] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.253012 8347 raft_consensus.cc:3037] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.253901 8347 raft_consensus.cc:2454] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601742aefcaa4d469c4587823fdc81ab in term 1.
I20260929 00:57:20.254208 8207 raft_consensus.cc:2454] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601742aefcaa4d469c4587823fdc81ab in term 1.
I20260929 00:57:20.254208 7999 leader_election.cc:302] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e09e2a7a38240b18c60e6dc81f7a2c0, 601742aefcaa4d469c4587823fdc81ab; no voters:
I20260929 00:57:20.254401 8429 raft_consensus.cc:2781] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:20.254688 8429 raft_consensus.cc:686] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 1 LEADER]: Becoming Leader. State: Replica: 601742aefcaa4d469c4587823fdc81ab, State: Running, Role: LEADER
I20260929 00:57:20.254941 8429 consensus_queue.cc:237] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.257337 7910 catalog_manager.cc:5495] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 601742aefcaa4d469c4587823fdc81ab (127.0.98.1). New cstate: current_term: 1 leader_uuid: "601742aefcaa4d469c4587823fdc81ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:57:20 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:20.300565 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.292005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36883 (local address 127.0.0.1:58120)
0929 00:57:20.292310 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.292328 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.292467 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.292765 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.292767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.293182 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:20.293291 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.293297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.294010 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.294013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.294640 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.294649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.295036 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.295038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.295106 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.295710 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:20.295721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:20.297297 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:20.299054 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:20.299062 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:20.299067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:20.299302 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:20.299564 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:20.299567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:20.299569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:20.299670 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:20.300074 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:20.300081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:20.300188 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.300275 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.300399 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":126,"threads_started":1}
I20260929 00:57:20.300582 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.292016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.0.98.62:36883)
0929 00:57:20.292258 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.292261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.292413 (+ 152us) server_negotiation.cc:407] Connection header received
0929 00:57:20.292547 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.292549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.292588 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.292683 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:20.293403 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.293917 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.294737 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.294935 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.295010 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.297450 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:20.297453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:20.297454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:20.297475 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:20.298953 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:20.299388 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:20.299391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:20.299392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:20.299460 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:20.299749 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:20.299751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:20.299752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:20.299986 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:20.300060 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.300329 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.300454 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":159}
I20260929 00:57:20.304551 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58120:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:57:20.305215 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:20.309811 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 71b41860b44e407ba3a1ae8cf5177a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db9efcee046647c6ab41048cd97d79b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:57:20.309912 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 71b41860b44e407ba3a1ae8cf5177a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db9efcee046647c6ab41048cd97d79b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:57:20.310348 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b41860b44e407ba3a1ae8cf5177a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.310537 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 71b41860b44e407ba3a1ae8cf5177a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db9efcee046647c6ab41048cd97d79b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:57:20.310902 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b41860b44e407ba3a1ae8cf5177a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.313009 8421 tablet_bootstrap.cc:492] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: Bootstrap starting.
I20260929 00:57:20.313490 8422 tablet_bootstrap.cc:492] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Bootstrap starting.
I20260929 00:57:20.313582 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b41860b44e407ba3a1ae8cf5177a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.313978 8421 tablet_bootstrap.cc:654] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.314489 8422 tablet_bootstrap.cc:654] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.315235 8421 tablet_bootstrap.cc:492] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: No bootstrap required, opened a new log
I20260929 00:57:20.315330 8421 ts_tablet_manager.cc:1397] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:20.315687 8421 raft_consensus.cc:348] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.315778 8421 raft_consensus.cc:374] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.315802 8421 raft_consensus.cc:729] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.315814 8422 tablet_bootstrap.cc:492] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: No bootstrap required, opened a new log
I20260929 00:57:20.315896 8422 ts_tablet_manager.cc:1397] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:20.315950 8421 consensus_queue.cc:260] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.316195 8427 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.316339 8427 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:57:20.316291 8422 raft_consensus.cc:348] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.316407 8422 raft_consensus.cc:374] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.316439 8422 raft_consensus.cc:729] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e09e2a7a38240b18c60e6dc81f7a2c0, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.316267 8421 ts_tablet_manager.cc:1428] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:57:20.316553 8422 consensus_queue.cc:260] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.316608 8424 tablet_bootstrap.cc:492] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: Bootstrap starting.
I20260929 00:57:20.316807 8422 ts_tablet_manager.cc:1428] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:20.317519 8424 tablet_bootstrap.cc:654] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.317444 8428 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.317584 8428 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:57:20.318821 8424 tablet_bootstrap.cc:492] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: No bootstrap required, opened a new log
I20260929 00:57:20.318894 8424 ts_tablet_manager.cc:1397] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:20.319298 8424 raft_consensus.cc:348] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.319387 8424 raft_consensus.cc:374] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.319409 8424 raft_consensus.cc:729] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601742aefcaa4d469c4587823fdc81ab, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.319541 8424 consensus_queue.cc:260] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.319804 8424 ts_tablet_manager.cc:1428] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:57:20.319751 8429 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.319994 8429 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:57:20.352788 8427 raft_consensus.cc:482] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:20.352916 8427 raft_consensus.cc:504] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.353359 8427 leader_election.cc:288] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.353766 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b41860b44e407ba3a1ae8cf5177a12" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601742aefcaa4d469c4587823fdc81ab" is_pre_election: true
I20260929 00:57:20.353919 8051 raft_consensus.cc:2454] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 0.
I20260929 00:57:20.353875 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b41860b44e407ba3a1ae8cf5177a12" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" is_pre_election: true
I20260929 00:57:20.354048 8347 raft_consensus.cc:2454] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 0.
I20260929 00:57:20.354254 8139 leader_election.cc:302] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601742aefcaa4d469c4587823fdc81ab, ecbf3738948e4892b6c53eb9a6d12fe4; no voters:
I20260929 00:57:20.354461 8427 raft_consensus.cc:2781] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:20.354539 8427 raft_consensus.cc:482] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:20.354592 8427 raft_consensus.cc:3037] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.355558 8427 raft_consensus.cc:504] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.355876 8427 leader_election.cc:288] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 election: Requested vote from peers 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.356172 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b41860b44e407ba3a1ae8cf5177a12" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601742aefcaa4d469c4587823fdc81ab"
I20260929 00:57:20.356287 8051 raft_consensus.cc:3037] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.356256 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b41860b44e407ba3a1ae8cf5177a12" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0"
I20260929 00:57:20.356389 8347 raft_consensus.cc:3037] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.357194 8347 raft_consensus.cc:2454] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 1.
I20260929 00:57:20.357220 8051 raft_consensus.cc:2454] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 1.
I20260929 00:57:20.357455 8139 leader_election.cc:302] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e09e2a7a38240b18c60e6dc81f7a2c0, ecbf3738948e4892b6c53eb9a6d12fe4; no voters:
I20260929 00:57:20.357666 8427 raft_consensus.cc:2781] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:20.357790 8427 raft_consensus.cc:686] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 LEADER]: Becoming Leader. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Running, Role: LEADER
I20260929 00:57:20.357923 8427 consensus_queue.cc:237] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.358597 8431 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: TxnStatusTablet state changed. Reason: New leader ecbf3738948e4892b6c53eb9a6d12fe4. Latest consensus state: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.358722 8431 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:57:20.358995 8464 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:57:20.359184 8464 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:57:20.359608 7909 catalog_manager.cc:5495] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:20.370334 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.366652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38443 (local address 127.0.0.1:36582)
0929 00:57:20.366917 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.366929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.367018 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.367400 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.367403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.367413 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:57:20.367525 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.367531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.368509 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.368511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.369117 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.369124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.369676 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.369678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.369738 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.369779 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.370060 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.370070 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.370115 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.370225 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260929 00:57:20.370343 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.366771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.0.98.2:38443)
0929 00:57:20.366902 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.366906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.366987 (+ 81us) server_negotiation.cc:407] Connection header received
0929 00:57:20.367124 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.367127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.367182 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.367267 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:20.367633 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.368409 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.369226 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.369562 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.369667 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.369899 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.369979 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.370053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.370136 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.370215 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:57:20.372960 8051 raft_consensus.cc:1261] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Refusing update from remote peer ecbf3738948e4892b6c53eb9a6d12fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.372960 8347 raft_consensus.cc:1261] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Refusing update from remote peer ecbf3738948e4892b6c53eb9a6d12fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.373368 8431 consensus_queue.cc:1035] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.373533 8427 consensus_queue.cc:1035] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.374818 8428 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: TxnStatusTablet state changed. Reason: New leader ecbf3738948e4892b6c53eb9a6d12fe4. Latest consensus state: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.374959 8428 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:57:20.375730 8429 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: TxnStatusTablet state changed. Reason: New leader ecbf3738948e4892b6c53eb9a6d12fe4. Latest consensus state: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.375845 8429 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:57:20.376336 8431 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.376434 8431 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:57:20.376449 8434 mvcc.cc:205] Tried to move back new op lower bound from 7334475531763617792 to 7334475531707572224. Current Snapshot: MvccSnapshot[applied={T|T < 7334475531763617792}]
I20260929 00:57:20.378860 8428 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.378978 8428 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P 4e09e2a7a38240b18c60e6dc81f7a2c0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 00:57:20.380354 8254 tablet.cc:2369] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:20.380403 8427 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.380503 8427 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P ecbf3738948e4892b6c53eb9a6d12fe4: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:57:20.382166 8436 mvcc.cc:205] Tried to move back new op lower bound from 7334475531763617792 to 7334475531707572224. Current Snapshot: MvccSnapshot[applied={T|T < 7334475531763617792}]
I20260929 00:57:20.382382 8429 tablet_replica.cc:439] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } } }
I20260929 00:57:20.382478 8429 tablet_replica.cc:442] T 71b41860b44e407ba3a1ae8cf5177a12 P 601742aefcaa4d469c4587823fdc81ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:57:20.385663 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.381500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.0.98.1:33221)
0929 00:57:20.381617 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.381622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.381636 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:57:20.381744 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.381746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.381809 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.381910 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:20.382310 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.383024 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.384124 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.384411 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.384527 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.384813 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.385265 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.385396 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.385463 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.385522 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 00:57:20.396798 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.393168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57561 (local address 127.0.98.2:38443)
0929 00:57:20.393306 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.393316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.393456 (+ 140us) server_negotiation.cc:407] Connection header received
0929 00:57:20.393578 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.393580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.393634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.393696 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:20.393950 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.394486 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.395717 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.395920 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.395998 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.396313 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.396376 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.396423 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.396563 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.396664 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:57:20.397038 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.393021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38443 (local address 127.0.98.1:57561)
0929 00:57:20.393368 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.393384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.393488 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.393724 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.393728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.393735 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:57:20.393854 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.393859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.394584 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.394588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.395598 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.395609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.396026 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.396030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.396134 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.396183 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.396450 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.396456 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.396541 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.396620 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":155,"threads_started":1}
I20260929 00:57:20.403918 8347 raft_consensus.cc:1261] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Refusing update from remote peer 601742aefcaa4d469c4587823fdc81ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.404211 8207 raft_consensus.cc:1261] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Refusing update from remote peer 601742aefcaa4d469c4587823fdc81ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.404783 8452 consensus_queue.cc:1035] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.405021 8429 consensus_queue.cc:1035] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.430771 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.426036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51421 (local address 127.0.98.1:33221)
0929 00:57:20.426160 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.426164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.426944 (+ 780us) server_negotiation.cc:407] Connection header received
0929 00:57:20.427127 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.427131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.427176 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.427219 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:20.427536 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.428316 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.429551 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.429888 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.429951 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.430335 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.430410 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.430448 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.430584 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.430646 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:57:20.431171 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.425781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33221 (local address 127.0.98.2:51421)
0929 00:57:20.426871 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.426885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.427014 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.427342 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.427345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.427352 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:57:20.427444 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.427448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.428445 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.428448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.429424 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.429434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.430051 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.430054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.430134 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.430200 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.430501 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.430505 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.430956 (+ 451us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.431034 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":98,"threads_started":1}
I20260929 00:57:20.461521 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:20.461699 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:20.463919 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: Sending DeleteTablet for 3 replicas of tablet 15612635644e498b809fd024bee15aa9
I20260929 00:57:20.464572 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 15612635644e498b809fd024bee15aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36562
I20260929 00:57:20.464601 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 15612635644e498b809fd024bee15aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260929 00:57:20.464752 8445 tablet_replica.cc:330] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: stopping tablet replica
I20260929 00:57:20.464797 8446 tablet_replica.cc:330] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: stopping tablet replica
I20260929 00:57:20.464876 8445 raft_consensus.cc:2229] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:20.464896 8446 raft_consensus.cc:2229] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:20.464993 8446 raft_consensus.cc:2258] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.464993 8445 raft_consensus.cc:2258] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.465168 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 15612635644e498b809fd024bee15aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42110
I20260929 00:57:20.465345 8444 tablet_replica.cc:330] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: stopping tablet replica
I20260929 00:57:20.465471 8444 raft_consensus.cc:2229] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:20.465806 8444 raft_consensus.cc:2258] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.466360 8446 ts_tablet_manager.cc:1905] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.466392 8445 ts_tablet_manager.cc:1905] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.466502 8444 ts_tablet_manager.cc:1905] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.469159 8445 ts_tablet_manager.cc:1918] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:57:20.469162 8444 ts_tablet_manager.cc:1918] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:57:20.469233 8444 log.cc:1196] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/15612635644e498b809fd024bee15aa9
I20260929 00:57:20.469233 8445 log.cc:1196] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/15612635644e498b809fd024bee15aa9
I20260929 00:57:20.469244 8446 ts_tablet_manager.cc:1918] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:57:20.469311 8446 log.cc:1196] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/15612635644e498b809fd024bee15aa9
I20260929 00:57:20.469499 8444 ts_tablet_manager.cc:1939] T 15612635644e498b809fd024bee15aa9 P 601742aefcaa4d469c4587823fdc81ab: Deleting consensus metadata
I20260929 00:57:20.469499 8445 ts_tablet_manager.cc:1939] T 15612635644e498b809fd024bee15aa9 P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting consensus metadata
I20260929 00:57:20.469570 8446 ts_tablet_manager.cc:1939] T 15612635644e498b809fd024bee15aa9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting consensus metadata
I20260929 00:57:20.470348 7895 catalog_manager.cc:4841] TS 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057): tablet 15612635644e498b809fd024bee15aa9 (table test-table [id=9093d77835f64264a8c9756a678544ff]) successfully deleted
I20260929 00:57:20.470363 7897 catalog_manager.cc:4841] TS ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2:38443): tablet 15612635644e498b809fd024bee15aa9 (table test-table [id=9093d77835f64264a8c9756a678544ff]) successfully deleted
I20260929 00:57:20.471331 7895 catalog_manager.cc:4841] TS 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221): tablet 15612635644e498b809fd024bee15aa9 (table test-table [id=9093d77835f64264a8c9756a678544ff]) successfully deleted
Sep 29 00:57:20 dist-test-slave-30s6 krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:57:20 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:20 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:57:20.503582 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:20.495463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.0.98.62:36883)
0929 00:57:20.495596 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.495601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.495882 (+ 281us) server_negotiation.cc:407] Connection header received
0929 00:57:20.496012 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.496016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.496076 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.496189 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:20.497114 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.497948 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.499204 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.499594 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.499708 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.503443 (+ 3735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58132: BlockingRecv error: recv got EOF from 127.0.0.1:58132 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 00:57:20 dist-test-slave-30s6 krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 00:57:20.515851 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.505165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.0.98.62:36883)
0929 00:57:20.505280 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.505284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.505704 (+ 420us) server_negotiation.cc:407] Connection header received
0929 00:57:20.505811 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.505814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.505867 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.505959 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:20.506694 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.507449 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.508579 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.508894 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.509010 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.511931 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:20.511936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:20.511939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:20.511965 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:20.513695 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:20.514267 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:20.514271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:20.514273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:20.514360 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:20.514709 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:20.514713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:20.514714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:20.515107 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:20.515252 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.515544 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.515667 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:57:20.518994 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:20.519469 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:20.524622 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 2949000b0a8c46b39fbc2546b3f5b6e9 (DEFAULT_TABLE table=test-table [id=feb23650a14342808cd68d4b689b08f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.524641 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 2949000b0a8c46b39fbc2546b3f5b6e9 (DEFAULT_TABLE table=test-table [id=feb23650a14342808cd68d4b689b08f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.524936 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2949000b0a8c46b39fbc2546b3f5b6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.524969 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2949000b0a8c46b39fbc2546b3f5b6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.527401 8424 tablet_bootstrap.cc:492] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: Bootstrap starting.
I20260929 00:57:20.527420 8421 tablet_bootstrap.cc:492] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: Bootstrap starting.
I20260929 00:57:20.528227 8421 tablet_bootstrap.cc:654] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.528312 8424 tablet_bootstrap.cc:654] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.529243 8421 tablet_bootstrap.cc:492] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: No bootstrap required, opened a new log
I20260929 00:57:20.529321 8421 ts_tablet_manager.cc:1397] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 00:57:20.529678 8421 raft_consensus.cc:348] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.529803 8421 raft_consensus.cc:374] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.529842 8421 raft_consensus.cc:729] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.529953 8421 consensus_queue.cc:260] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.530121 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 2949000b0a8c46b39fbc2546b3f5b6e9 (DEFAULT_TABLE table=test-table [id=feb23650a14342808cd68d4b689b08f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.530222 8421 ts_tablet_manager.cc:1428] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:20.530397 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2949000b0a8c46b39fbc2546b3f5b6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.533098 8422 tablet_bootstrap.cc:492] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Bootstrap starting.
I20260929 00:57:20.534001 8422 tablet_bootstrap.cc:654] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.535481 8424 tablet_bootstrap.cc:492] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: No bootstrap required, opened a new log
I20260929 00:57:20.535568 8424 ts_tablet_manager.cc:1397] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 00:57:20.536047 8424 raft_consensus.cc:348] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.536175 8424 raft_consensus.cc:374] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.536216 8424 raft_consensus.cc:729] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601742aefcaa4d469c4587823fdc81ab, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.536330 8424 consensus_queue.cc:260] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.536561 8424 ts_tablet_manager.cc:1428] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:20.537372 8422 tablet_bootstrap.cc:492] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: No bootstrap required, opened a new log
I20260929 00:57:20.537449 8422 ts_tablet_manager.cc:1397] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 00:57:20.537856 8422 raft_consensus.cc:348] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.537950 8422 raft_consensus.cc:374] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.537986 8422 raft_consensus.cc:729] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e09e2a7a38240b18c60e6dc81f7a2c0, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.538089 8422 consensus_queue.cc:260] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.538364 8422 ts_tablet_manager.cc:1428] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:20.579665 8427 raft_consensus.cc:482] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:20.579828 8427 raft_consensus.cc:504] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.580359 8427 leader_election.cc:288] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.580803 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2949000b0a8c46b39fbc2546b3f5b6e9" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601742aefcaa4d469c4587823fdc81ab" is_pre_election: true
I20260929 00:57:20.580876 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2949000b0a8c46b39fbc2546b3f5b6e9" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" is_pre_election: true
I20260929 00:57:20.580960 8051 raft_consensus.cc:2454] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 0.
I20260929 00:57:20.581032 8347 raft_consensus.cc:2454] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 0.
I20260929 00:57:20.581375 8139 leader_election.cc:302] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601742aefcaa4d469c4587823fdc81ab, ecbf3738948e4892b6c53eb9a6d12fe4; no voters:
I20260929 00:57:20.581599 8427 raft_consensus.cc:2781] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:20.581671 8427 raft_consensus.cc:482] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:20.581722 8427 raft_consensus.cc:3037] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.583040 8427 raft_consensus.cc:504] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.583400 8427 leader_election.cc:288] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 election: Requested vote from peers 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.583710 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2949000b0a8c46b39fbc2546b3f5b6e9" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601742aefcaa4d469c4587823fdc81ab"
I20260929 00:57:20.583835 8051 raft_consensus.cc:3037] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.583823 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2949000b0a8c46b39fbc2546b3f5b6e9" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0"
I20260929 00:57:20.583943 8347 raft_consensus.cc:3037] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.584995 8347 raft_consensus.cc:2454] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 1.
I20260929 00:57:20.584995 8051 raft_consensus.cc:2454] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 1.
I20260929 00:57:20.585299 8139 leader_election.cc:302] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e09e2a7a38240b18c60e6dc81f7a2c0, ecbf3738948e4892b6c53eb9a6d12fe4; no voters:
I20260929 00:57:20.585469 8427 raft_consensus.cc:2781] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:20.585606 8427 raft_consensus.cc:686] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 LEADER]: Becoming Leader. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Running, Role: LEADER
I20260929 00:57:20.585752 8427 consensus_queue.cc:237] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.587515 7910 catalog_manager.cc:5495] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:20.607708 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.604211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.0.98.2:38443)
0929 00:57:20.604338 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.604346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.604386 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:57:20.604530 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.604532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.604590 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.604702 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:20.604924 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.605410 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.606682 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.606890 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.606968 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.607256 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.607322 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.607373 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.607513 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.607552 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:57:20.609838 8347 raft_consensus.cc:1261] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Refusing update from remote peer ecbf3738948e4892b6c53eb9a6d12fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.609867 8051 raft_consensus.cc:1261] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Refusing update from remote peer ecbf3738948e4892b6c53eb9a6d12fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.610268 8427 consensus_queue.cc:1035] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.610410 8431 consensus_queue.cc:1035] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.618331 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:20.618498 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:20.620880 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: Sending DeleteTablet for 3 replicas of tablet 2949000b0a8c46b39fbc2546b3f5b6e9
I20260929 00:57:20.621587 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 2949000b0a8c46b39fbc2546b3f5b6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42110
I20260929 00:57:20.621779 8444 tablet_replica.cc:330] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: stopping tablet replica
I20260929 00:57:20.621933 8444 raft_consensus.cc:2229] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:20.622084 8444 raft_consensus.cc:2258] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.622288 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet 2949000b0a8c46b39fbc2546b3f5b6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260929 00:57:20.622400 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 2949000b0a8c46b39fbc2546b3f5b6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36562
I20260929 00:57:20.622551 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 2949000b0a8c46b39fbc2546b3f5b6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260929 00:57:20.622627 8446 tablet_replica.cc:330] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: stopping tablet replica
I20260929 00:57:20.622754 8446 raft_consensus.cc:2229] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:20.622885 8446 raft_consensus.cc:2258] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.622994 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:57:20.623162 8445 tablet_replica.cc:330] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: stopping tablet replica
I20260929 00:57:20.623281 8445 raft_consensus.cc:2229] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 LEADER]: Raft consensus shutting down.
W20260929 00:57:20.623431 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:20.623461 8445 raft_consensus.cc:2258] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.624213 8445 ts_tablet_manager.cc:1905] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.624311 8446 ts_tablet_manager.cc:1905] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.625049 8444 ts_tablet_manager.cc:1905] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.626860 8445 ts_tablet_manager.cc:1918] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:20.626901 8446 ts_tablet_manager.cc:1918] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:20.626957 8445 log.cc:1196] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/2949000b0a8c46b39fbc2546b3f5b6e9
I20260929 00:57:20.626988 8446 log.cc:1196] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/2949000b0a8c46b39fbc2546b3f5b6e9
I20260929 00:57:20.627210 8445 ts_tablet_manager.cc:1939] T 2949000b0a8c46b39fbc2546b3f5b6e9 P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting consensus metadata
I20260929 00:57:20.627238 8446 ts_tablet_manager.cc:1939] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting consensus metadata
I20260929 00:57:20.627652 8444 ts_tablet_manager.cc:1918] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:20.627723 8444 log.cc:1196] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/2949000b0a8c46b39fbc2546b3f5b6e9
I20260929 00:57:20.627949 8444 ts_tablet_manager.cc:1939] T 2949000b0a8c46b39fbc2546b3f5b6e9 P 601742aefcaa4d469c4587823fdc81ab: Deleting consensus metadata
I20260929 00:57:20.628871 7897 catalog_manager.cc:4841] TS ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2:38443): tablet 2949000b0a8c46b39fbc2546b3f5b6e9 (table test-table [id=feb23650a14342808cd68d4b689b08f5]) successfully deleted
I20260929 00:57:20.630436 7895 catalog_manager.cc:4841] TS 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221): tablet 2949000b0a8c46b39fbc2546b3f5b6e9 (table test-table [id=feb23650a14342808cd68d4b689b08f5]) successfully deleted
I20260929 00:57:20.631047 7895 catalog_manager.cc:4841] TS 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057): tablet 2949000b0a8c46b39fbc2546b3f5b6e9 (table test-table [id=feb23650a14342808cd68d4b689b08f5]) successfully deleted
W20260929 00:57:20.631291 7895 catalog_manager.cc:4808] TS 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057): delete failed for tablet 2949000b0a8c46b39fbc2546b3f5b6e9 because the tablet was not found. No further retry: Not found: Tablet not found: 2949000b0a8c46b39fbc2546b3f5b6e9
I20260929 00:57:20.634342 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 755ce439cd454f96b14156ae05941bcc (DEFAULT_TABLE table=test-table [id=112c1faadda04ddf9651afd0fcd25e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.634459 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 755ce439cd454f96b14156ae05941bcc (DEFAULT_TABLE table=test-table [id=112c1faadda04ddf9651afd0fcd25e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.634985 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 755ce439cd454f96b14156ae05941bcc (DEFAULT_TABLE table=test-table [id=112c1faadda04ddf9651afd0fcd25e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:20.635275 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755ce439cd454f96b14156ae05941bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.635854 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755ce439cd454f96b14156ae05941bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.637614 8422 tablet_bootstrap.cc:492] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: Bootstrap starting.
I20260929 00:57:20.637751 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755ce439cd454f96b14156ae05941bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:20.638548 8422 tablet_bootstrap.cc:654] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.640223 8424 tablet_bootstrap.cc:492] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: Bootstrap starting.
I20260929 00:57:20.640779 8421 tablet_bootstrap.cc:492] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: Bootstrap starting.
I20260929 00:57:20.641016 8424 tablet_bootstrap.cc:654] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.641619 8421 tablet_bootstrap.cc:654] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:20.642251 8424 tablet_bootstrap.cc:492] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: No bootstrap required, opened a new log
I20260929 00:57:20.642336 8424 ts_tablet_manager.cc:1397] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:20.642722 8424 raft_consensus.cc:348] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.642838 8424 raft_consensus.cc:374] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.642867 8424 raft_consensus.cc:729] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601742aefcaa4d469c4587823fdc81ab, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.642987 8424 consensus_queue.cc:260] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.643221 8424 ts_tablet_manager.cc:1428] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:20.644063 8421 tablet_bootstrap.cc:492] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: No bootstrap required, opened a new log
I20260929 00:57:20.644197 8421 ts_tablet_manager.cc:1397] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 00:57:20.644708 8421 raft_consensus.cc:348] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.644838 8421 raft_consensus.cc:374] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.644870 8421 raft_consensus.cc:729] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.645015 8421 consensus_queue.cc:260] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.645287 8421 ts_tablet_manager.cc:1428] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:57:20.645983 8422 tablet_bootstrap.cc:492] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: No bootstrap required, opened a new log
I20260929 00:57:20.646090 8422 ts_tablet_manager.cc:1397] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 00:57:20.646692 8422 raft_consensus.cc:348] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.646809 8422 raft_consensus.cc:374] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:20.646842 8422 raft_consensus.cc:729] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e09e2a7a38240b18c60e6dc81f7a2c0, State: Initialized, Role: FOLLOWER
I20260929 00:57:20.646960 8422 consensus_queue.cc:260] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.647207 8422 ts_tablet_manager.cc:1428] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:57:20.850953 8427 raft_consensus.cc:482] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:20.851110 8427 raft_consensus.cc:504] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.851536 8427 leader_election.cc:288] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.852044 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755ce439cd454f96b14156ae05941bcc" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601742aefcaa4d469c4587823fdc81ab" is_pre_election: true
I20260929 00:57:20.852078 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755ce439cd454f96b14156ae05941bcc" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" is_pre_election: true
I20260929 00:57:20.852221 8347 raft_consensus.cc:2454] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 0.
I20260929 00:57:20.852221 8051 raft_consensus.cc:2454] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 0.
I20260929 00:57:20.852525 8139 leader_election.cc:302] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601742aefcaa4d469c4587823fdc81ab, ecbf3738948e4892b6c53eb9a6d12fe4; no voters:
I20260929 00:57:20.852723 8427 raft_consensus.cc:2781] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:20.852785 8427 raft_consensus.cc:482] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:20.852826 8427 raft_consensus.cc:3037] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.853870 8427 raft_consensus.cc:504] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.854287 8427 leader_election.cc:288] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 election: Requested vote from peers 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221), 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057)
I20260929 00:57:20.854661 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755ce439cd454f96b14156ae05941bcc" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601742aefcaa4d469c4587823fdc81ab"
I20260929 00:57:20.854710 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755ce439cd454f96b14156ae05941bcc" candidate_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0"
I20260929 00:57:20.854813 8051 raft_consensus.cc:3037] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.854832 8347 raft_consensus.cc:3037] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:20.855722 8051 raft_consensus.cc:2454] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 1.
I20260929 00:57:20.855930 8347 raft_consensus.cc:2454] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbf3738948e4892b6c53eb9a6d12fe4 in term 1.
I20260929 00:57:20.856017 8139 leader_election.cc:302] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601742aefcaa4d469c4587823fdc81ab, ecbf3738948e4892b6c53eb9a6d12fe4; no voters:
I20260929 00:57:20.856292 8427 raft_consensus.cc:2781] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:20.856392 8427 raft_consensus.cc:686] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 LEADER]: Becoming Leader. State: Replica: ecbf3738948e4892b6c53eb9a6d12fe4, State: Running, Role: LEADER
I20260929 00:57:20.856544 8427 consensus_queue.cc:237] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } }
I20260929 00:57:20.858297 7907 catalog_manager.cc:5495] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecbf3738948e4892b6c53eb9a6d12fe4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:20.902762 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.899942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35195 (local address 127.0.98.2:38443)
0929 00:57:20.900055 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:20.900058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:20.900119 (+ 61us) server_negotiation.cc:407] Connection header received
0929 00:57:20.900219 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:20.900221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:20.900259 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:20.900323 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:20.900544 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.901124 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.901881 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.902082 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.902153 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.902398 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.902461 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.902517 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:20.902601 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 00:57:20.902639 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44}
I20260929 00:57:20.902793 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:20.899886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38443 (local address 127.0.98.2:35195)
0929 00:57:20.900043 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:20.900056 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:20.900140 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:20.900341 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:20.900343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:20.900349 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:57:20.900449 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.900453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.901213 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.901215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.901785 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:20.901792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:20.902170 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:20.902173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:20.902277 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:20.902301 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:57:20.902532 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:57:20.902537 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:57:20.902594 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 00:57:20.902658 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":14}
I20260929 00:57:20.909159 8347 raft_consensus.cc:1261] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Refusing update from remote peer ecbf3738948e4892b6c53eb9a6d12fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.909215 8051 raft_consensus.cc:1261] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Refusing update from remote peer ecbf3738948e4892b6c53eb9a6d12fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:20.909616 8440 consensus_queue.cc:1035] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601742aefcaa4d469c4587823fdc81ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.909775 8431 consensus_queue.cc:1035] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e09e2a7a38240b18c60e6dc81f7a2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:20.938911 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:20.939057 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:20.940867 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 325267e44ac446f49041080a48172c1c: Sending DeleteTablet for 3 replicas of tablet 755ce439cd454f96b14156ae05941bcc
I20260929 00:57:20.941488 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 755ce439cd454f96b14156ae05941bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42110
I20260929 00:57:20.941504 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 755ce439cd454f96b14156ae05941bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260929 00:57:20.941637 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 755ce439cd454f96b14156ae05941bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36562
I20260929 00:57:20.941695 8444 tablet_replica.cc:330] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: stopping tablet replica
I20260929 00:57:20.941704 8446 tablet_replica.cc:330] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: stopping tablet replica
I20260929 00:57:20.941811 8444 raft_consensus.cc:2229] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:20.941811 8446 raft_consensus.cc:2229] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:20.941949 8444 raft_consensus.cc:2258] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.941951 8446 raft_consensus.cc:2258] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.942142 8445 tablet_replica.cc:330] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: stopping tablet replica
I20260929 00:57:20.942287 8445 raft_consensus.cc:2229] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:20.942471 8445 raft_consensus.cc:2258] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:20.942780 8446 ts_tablet_manager.cc:1905] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.943169 8444 ts_tablet_manager.cc:1905] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.944785 8446 ts_tablet_manager.cc:1918] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:57:20.944882 8446 log.cc:1196] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/755ce439cd454f96b14156ae05941bcc
I20260929 00:57:20.945067 8444 ts_tablet_manager.cc:1918] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:57:20.945137 8444 log.cc:1196] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/755ce439cd454f96b14156ae05941bcc
I20260929 00:57:20.945142 8446 ts_tablet_manager.cc:1939] T 755ce439cd454f96b14156ae05941bcc P 4e09e2a7a38240b18c60e6dc81f7a2c0: Deleting consensus metadata
I20260929 00:57:20.945382 8444 ts_tablet_manager.cc:1939] T 755ce439cd454f96b14156ae05941bcc P 601742aefcaa4d469c4587823fdc81ab: Deleting consensus metadata
I20260929 00:57:20.945838 7895 catalog_manager.cc:4841] TS 4e09e2a7a38240b18c60e6dc81f7a2c0 (127.0.98.3:40057): tablet 755ce439cd454f96b14156ae05941bcc (table test-table [id=112c1faadda04ddf9651afd0fcd25e59]) successfully deleted
I20260929 00:57:20.946033 7895 catalog_manager.cc:4841] TS 601742aefcaa4d469c4587823fdc81ab (127.0.98.1:33221): tablet 755ce439cd454f96b14156ae05941bcc (table test-table [id=112c1faadda04ddf9651afd0fcd25e59]) successfully deleted
I20260929 00:57:20.946406 8445 ts_tablet_manager.cc:1905] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:20.946491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 7982
I20260929 00:57:20.948770 8445 ts_tablet_manager.cc:1918] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:57:20.948836 8445 log.cc:1196] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/755ce439cd454f96b14156ae05941bcc
I20260929 00:57:20.949098 8445 ts_tablet_manager.cc:1939] T 755ce439cd454f96b14156ae05941bcc P ecbf3738948e4892b6c53eb9a6d12fe4: Deleting consensus metadata
I20260929 00:57:20.949752 7897 catalog_manager.cc:4841] TS ecbf3738948e4892b6c53eb9a6d12fe4 (127.0.98.2:38443): tablet 755ce439cd454f96b14156ae05941bcc (table test-table [id=112c1faadda04ddf9651afd0fcd25e59]) successfully deleted
I20260929 00:57:20.956256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 8122
I20260929 00:57:20.963047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 8262
I20260929 00:57:20.968818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 7878
2026-09-29T00:57:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3218 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:21 dist-test-slave-30s6 krb5kdc[8506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:21 dist-test-slave-30s6 krb5kdc[8506](info): set up 2 sockets
Sep 29 00:57:21 dist-test-slave-30s6 krb5kdc[8506](info): commencing operation
krb5kdc: starting...
W20260929 00:57:22.019217 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:22 dist-test-slave-30s6 krb5kdc[8506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:57:22.146080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:22.256907 8519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:22.257194 8519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:22.257234 8519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:22.261322 8519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:22.261442 8519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:22.261478 8519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:22.261518 8519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:22.261547 8519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:22.267004 8519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:22.268167 8519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:22.269237 8519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:22.277974 8525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:22.277987 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:22.277995 8524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:22.278380 8519 server_base.cc:1034] running on GCE node
I20260929 00:57:22.279006 8519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:22.280134 8519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:22.281322 8519 hybrid_clock.cc:648] HybridClock initialized: now 1790643442281289 us; error 45 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T00:57:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1324 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:22 dist-test-slave-30s6 krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:22 dist-test-slave-30s6 krb5kdc[8532](info): set up 2 sockets
Sep 29 00:57:22 dist-test-slave-30s6 krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260929 00:57:23.358971 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:23 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:57:23.481096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:23.592826 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:23.593082 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:23.593150 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:23.597167 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:23.597222 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:23.597242 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:23.597298 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:23.597332 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:23.602581 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:23.603688 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:23.604734 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:23.613267 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:23.613260 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:23.613260 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:23.613343 8545 server_base.cc:1034] running on GCE node
I20260929 00:57:23.614259 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:23.615367 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:23.616556 8545 hybrid_clock.cc:648] HybridClock initialized: now 1790643443616528 us; error 43 us; skew 500 ppm
Sep 29 00:57:23 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:23.619220 8545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:57:23.620146 8545 webserver.cc:460] Webserver started at http://127.0.98.62:33879/ using document root <none> and password file <none>
I20260929 00:57:23.620711 8545 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:23.620802 8545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:23.621040 8545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:23.623134 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "6ad75345c4d84781a840405d1212b571"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-30s6"
server_key: "bf4c919e1aa9952fa7698322a6c70efe"
server_key_iv: "c8bec158cabc6e27342280b77899cc30"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.623693 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ad75345c4d84781a840405d1212b571"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-30s6"
server_key: "bf4c919e1aa9952fa7698322a6c70efe"
server_key_iv: "c8bec158cabc6e27342280b77899cc30"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.627322 8545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 00:57:23.629649 8560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:23.630976 8545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:57:23.631116 8545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "6ad75345c4d84781a840405d1212b571"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-30s6"
server_key: "bf4c919e1aa9952fa7698322a6c70efe"
server_key_iv: "c8bec158cabc6e27342280b77899cc30"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.631233 8545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:23.650056 8545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:23.653472 8545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:23.653715 8545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:23.661078 8545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39635
I20260929 00:57:23.661103 8612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39635 every 8 connection(s)
I20260929 00:57:23.662022 8545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:23.664798 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:23.667266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 8545
I20260929 00:57:23.667366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:23.667675 392 external_mini_cluster.cc:1428] Setting key 9566bbb43083bf058d43a9088ced24d4
I20260929 00:57:23.670737 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Bootstrap starting.
I20260929 00:57:23.672935 8613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:23.673655 8613 log.cc:824] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:23.675644 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: No bootstrap required, opened a new log
Sep 29 00:57:23 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:23.679363 8613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260929 00:57:23.679665 8613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:23.679745 8613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad75345c4d84781a840405d1212b571, State: Initialized, Role: FOLLOWER
I20260929 00:57:23.680245 8613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260929 00:57:23.680374 8613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:23.680450 8613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:23.680546 8613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:23.681636 8613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260929 00:57:23.681981 8613 leader_election.cc:302] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ad75345c4d84781a840405d1212b571; no voters:
I20260929 00:57:23.682350 8613 leader_election.cc:288] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:23.682427 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:23.669013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.0.98.62:39635)
0929 00:57:23.669438 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:23.669453 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:23.669489 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:57:23.670106 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:23.670123 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:23.670522 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:23.670857 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:23.671655 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:23.672640 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:23.673549 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:23.674078 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:23.674269 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:23.676949 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:23.676954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:23.676967 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:23.676994 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:23.679124 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:23.679665 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:23.679673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:23.679680 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:23.679854 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:23.680157 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:23.680159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:23.680160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:23.680704 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:23.680856 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:23.681144 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 00:57:23.681448 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260929 00:57:23.682439 8618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:23.682989 8618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad75345c4d84781a840405d1212b571, State: Running, Role: LEADER
I20260929 00:57:23.683284 8618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260929 00:57:23.683633 8613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:23.684439 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad75345c4d84781a840405d1212b571. Latest consensus state: current_term: 1 leader_uuid: "6ad75345c4d84781a840405d1212b571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } } }
I20260929 00:57:23.684561 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:23.684937 8627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:23.685256 8619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ad75345c4d84781a840405d1212b571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } } }
I20260929 00:57:23.685348 8619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:23.687915 8627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:23.693630 8627 catalog_manager.cc:1347] Generated new cluster ID: 82edf90d68594bfebbd40b33c934ced5
I20260929 00:57:23.693703 8627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:23.705644 8627 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:23.706982 8627 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:23.714596 8627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Generated new TSK 0
I20260929 00:57:23.715399 8627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:57:23.743489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:23.855243 8639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:23.855544 8639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:23.855618 8639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:23.859726 8639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:23.859820 8639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:23.859920 8639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:23.865221 8639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:23.866429 8639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:23.867491 8639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:23.877468 8644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:23.877497 8639 server_base.cc:1034] running on GCE node
W20260929 00:57:23.877468 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:23.877525 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:23.878252 8639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:23.878916 8639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:23.880102 8639 hybrid_clock.cc:648] HybridClock initialized: now 1790643443880073 us; error 62 us; skew 500 ppm
Sep 29 00:57:23 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:23.882952 8639 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:57:23.883967 8639 webserver.cc:460] Webserver started at http://127.0.98.1:39955/ using document root <none> and password file <none>
I20260929 00:57:23.884574 8639 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:23.884671 8639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:23.884915 8639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:23.887167 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "b66397d3027a4243a44a72c5c3178d90"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-30s6"
server_key: "62f6e2c9d243c780e5b4a669c41a5ae8"
server_key_iv: "5890065bfa8b7d5d590e29f210a9a811"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.887758 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b66397d3027a4243a44a72c5c3178d90"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-30s6"
server_key: "62f6e2c9d243c780e5b4a669c41a5ae8"
server_key_iv: "5890065bfa8b7d5d590e29f210a9a811"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.891665 8639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:57:23.894330 8654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:23.895586 8639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:23.895725 8639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "b66397d3027a4243a44a72c5c3178d90"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-30s6"
server_key: "62f6e2c9d243c780e5b4a669c41a5ae8"
server_key_iv: "5890065bfa8b7d5d590e29f210a9a811"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.895850 8639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:23.912433 8639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:23.915527 8639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:23.915727 8639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:23.916361 8639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:23.917167 8639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:23.917239 8639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:23.917310 8639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:23.917354 8639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:23.926507 8639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44027
I20260929 00:57:23.926541 8767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44027 every 8 connection(s)
I20260929 00:57:23.927409 8639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:23.930055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 8639
I20260929 00:57:23.930228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:23.930558 392 external_mini_cluster.cc:1428] Setting key 48dcc8e3f869edaacf9e8c43ee3070c2
Sep 29 00:57:23 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 00:57:23.944003 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:23.929437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53429 (local address 127.0.98.62:39635)
0929 00:57:23.929580 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:23.929584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:23.930613 (+ 1029us) server_negotiation.cc:407] Connection header received
0929 00:57:23.932394 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:23.932398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:23.932456 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:23.932562 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:23.934051 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:23.934700 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:23.936646 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:23.936948 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:23.937052 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:23.940112 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:23.940116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:23.940118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:23.940142 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:23.941649 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:23.942231 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:23.942235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:23.942236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:23.942349 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:23.942806 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:23.942810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:23.942812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:23.943050 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:23.943162 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:23.943758 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 00:57:23.943882 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:57:23.945276 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:23.929838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:53429)
0929 00:57:23.930455 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:23.930501 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:23.932153 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:23.932766 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:23.932773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:23.933135 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:23.933819 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:23.933841 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:23.934834 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:23.934841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:23.936488 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:23.936505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:23.937137 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:23.937144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:23.937395 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:23.938116 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:23.938144 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:23.939917 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:23.941788 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:23.941794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:23.941812 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:23.942097 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:23.942527 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:23.942530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:23.942532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:23.942699 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:23.943160 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:23.943167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:23.943470 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 00:57:23.943706 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 00:57:23.944115 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":160,"threads_started":1}
I20260929 00:57:23.946889 8768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39635
I20260929 00:57:23.947156 8768 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:23.947783 8768 heartbeater.cc:511] Master 127.0.98.62:39635 requested a full tablet report, sending...
I20260929 00:57:23.949469 8577 ts_manager.cc:194] Registered new tserver with Master: b66397d3027a4243a44a72c5c3178d90 (127.0.98.1:44027)
I20260929 00:57:23.950788 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:57:23.962456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:24.073673 8773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:24.073922 8773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:24.074002 8773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:24.080904 8773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:24.080996 8773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:24.081099 8773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:24.090400 8773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:24.091806 8773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:24.093187 8773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:24.105847 8778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:24.105932 8773 server_base.cc:1034] running on GCE node
W20260929 00:57:24.106560 8781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.106621 8779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:24.107252 8773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:24.108196 8773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:24.109392 8773 hybrid_clock.cc:648] HybridClock initialized: now 1790643444109350 us; error 61 us; skew 500 ppm
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:24.113412 8773 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:57:24.114667 8773 webserver.cc:460] Webserver started at http://127.0.98.2:36249/ using document root <none> and password file <none>
I20260929 00:57:24.115425 8773 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:24.115513 8773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:24.115754 8773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:24.118465 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "76fbbccb606547d0bff3e19eed046330"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-30s6"
server_key: "2d476a722e230d304e97b6a3fbdf0fb7"
server_key_iv: "b92ec4ff0bb45e8dcf7ee1438817e328"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.119164 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76fbbccb606547d0bff3e19eed046330"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-30s6"
server_key: "2d476a722e230d304e97b6a3fbdf0fb7"
server_key_iv: "b92ec4ff0bb45e8dcf7ee1438817e328"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.124362 8773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 00:57:24.126958 8788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.128153 8773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:24.128314 8773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "76fbbccb606547d0bff3e19eed046330"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-30s6"
server_key: "2d476a722e230d304e97b6a3fbdf0fb7"
server_key_iv: "b92ec4ff0bb45e8dcf7ee1438817e328"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.128459 8773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:24.158442 8773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:24.161852 8773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:24.162060 8773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:24.162693 8773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:24.163690 8773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:24.163769 8773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.163839 8773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:24.163877 8773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.173028 8773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46027
I20260929 00:57:24.173050 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46027 every 8 connection(s)
I20260929 00:57:24.173916 8773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:24.179288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 8773
I20260929 00:57:24.179437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:24.179767 392 external_mini_cluster.cc:1428] Setting key 076d40580409271a64bd9c89d1f5259d
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:24.189550 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.175810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60039 (local address 127.0.98.62:39635)
0929 00:57:24.175947 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:24.175957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:24.176781 (+ 824us) server_negotiation.cc:407] Connection header received
0929 00:57:24.178143 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:24.178146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:24.178208 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:24.178285 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:24.179806 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.180405 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.181694 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.182112 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.182264 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.185466 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:24.185472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:24.185477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:24.185512 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:24.187198 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:24.187842 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:24.187845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:24.187846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:24.187922 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:24.188342 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:24.188346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:24.188348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:24.188572 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:24.188645 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:24.189325 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 00:57:24.189433 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:57:24.190693 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.176194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:60039)
0929 00:57:24.176617 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:24.176660 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:24.177933 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:24.178513 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:24.178525 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:24.178923 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:24.179632 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.179654 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.180555 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.180559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.181554 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.181566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.182292 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.182299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.182526 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.183324 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:24.183350 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:24.185262 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:24.187334 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:24.187343 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:24.187362 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:24.187719 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:24.188028 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:24.188034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:24.188036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:24.188226 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:24.188696 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:24.188705 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:24.189044 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 00:57:24.189285 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 00:57:24.189671 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
I20260929 00:57:24.191946 8902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39635
I20260929 00:57:24.192219 8902 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:57:24.192780 8902 heartbeater.cc:511] Master 127.0.98.62:39635 requested a full tablet report, sending...
I20260929 00:57:24.193885 8577 ts_manager.cc:194] Registered new tserver with Master: 76fbbccb606547d0bff3e19eed046330 (127.0.98.2:46027)
I20260929 00:57:24.194540 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:57:24.209610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:24.321987 8907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:24.322427 8907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:24.322525 8907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:24.326668 8907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:24.326764 8907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:24.326867 8907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:24.332324 8907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:24.333436 8907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:24.334633 8907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:24.344270 8913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.344291 8912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:24.344383 8907 server_base.cc:1034] running on GCE node
W20260929 00:57:24.344297 8915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:24.345031 8907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:24.345727 8907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:24.346915 8907 hybrid_clock.cc:648] HybridClock initialized: now 1790643444346895 us; error 33 us; skew 500 ppm
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:24.349695 8907 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:57:24.350687 8907 webserver.cc:460] Webserver started at http://127.0.98.3:44389/ using document root <none> and password file <none>
I20260929 00:57:24.351255 8907 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:24.351382 8907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:24.351614 8907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:24.353638 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "14cbc2f9d8424f11bccf71a5f6540622"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-30s6"
server_key: "c368ce6ecc5f9d893a2182fc0a8cf6b8"
server_key_iv: "a251bd5838e055a368b3a4c5fdec0893"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.354252 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14cbc2f9d8424f11bccf71a5f6540622"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-30s6"
server_key: "c368ce6ecc5f9d893a2182fc0a8cf6b8"
server_key_iv: "a251bd5838e055a368b3a4c5fdec0893"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.357924 8907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:24.360385 8922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.361582 8907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 00:57:24.361723 8907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "14cbc2f9d8424f11bccf71a5f6540622"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-30s6"
server_key: "c368ce6ecc5f9d893a2182fc0a8cf6b8"
server_key_iv: "a251bd5838e055a368b3a4c5fdec0893"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.361837 8907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:24.383042 8907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:24.386343 8907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:24.386565 8907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:24.387164 8907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:24.388005 8907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:24.388087 8907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.388180 8907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:24.388230 8907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.397056 8907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45457
I20260929 00:57:24.397078 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45457 every 8 connection(s)
I20260929 00:57:24.397964 8907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:24.406405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 8907
I20260929 00:57:24.406558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:24.406847 392 external_mini_cluster.cc:1428] Setting key e942e444e675b7a3100ba8d620a6dc92
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:24.412289 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.399966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52449 (local address 127.0.98.62:39635)
0929 00:57:24.400103 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:24.400106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:24.400930 (+ 824us) server_negotiation.cc:407] Connection header received
0929 00:57:24.402377 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:24.402380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:24.402436 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:24.402518 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:24.403678 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.404215 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.405353 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.405578 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.405692 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.408526 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:24.408529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:24.408531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:24.408556 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:24.410077 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:24.410743 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:24.410746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:24.410748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:24.410833 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:24.411193 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:24.411196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:24.411198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:24.411397 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:24.411506 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:24.412014 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 00:57:24.412134 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:57:24.413121 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.400341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:52449)
0929 00:57:24.400784 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:24.400829 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:24.402102 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:24.402688 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:24.402695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:24.403048 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:24.403529 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.403542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.404355 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.404359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.405217 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.405226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.405730 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.405734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.405905 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.406656 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:24.406680 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:24.408367 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:24.410249 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:24.410254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:24.410271 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:24.410640 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:24.410934 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:24.410937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:24.410939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:24.411087 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:24.411503 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:24.411510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:24.411765 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 00:57:24.411962 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 00:57:24.412291 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260929 00:57:24.414233 9036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39635
I20260929 00:57:24.414496 9036 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:24.414980 9036 heartbeater.cc:511] Master 127.0.98.62:39635 requested a full tablet report, sending...
I20260929 00:57:24.415997 8577 ts_manager.cc:194] Registered new tserver with Master: 14cbc2f9d8424f11bccf71a5f6540622 (127.0.98.3:45457)
I20260929 00:57:24.416579 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52449
I20260929 00:57:24.420837 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 00:57:24.498443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 8639
I20260929 00:57:24.504230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 8773
I20260929 00:57:24.509344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 8907
I20260929 00:57:24.514595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 8545
I20260929 00:57:24.520792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39635
--webserver_interface=127.0.98.62
--webserver_port=33879
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:24.632336 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:24.632606 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:24.632647 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:24.636729 9045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:24.636791 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:24.636811 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:24.636828 9045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:24.636843 9045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:24.642096 9045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:24.643185 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:24.644182 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:24.653040 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.653035 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.653040 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:24.653101 9045 server_base.cc:1034] running on GCE node
I20260929 00:57:24.653911 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:24.655102 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:24.656273 9045 hybrid_clock.cc:648] HybridClock initialized: now 1790643444656241 us; error 46 us; skew 500 ppm
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:24.659145 9045 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:57:24.660013 9045 webserver.cc:460] Webserver started at http://127.0.98.62:33879/ using document root <none> and password file <none>
I20260929 00:57:24.660579 9045 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:24.660674 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:24.664445 9045 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 00:57:24.666527 9060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.667721 9045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 00:57:24.667860 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "6ad75345c4d84781a840405d1212b571"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-30s6"
server_key: "bf4c919e1aa9952fa7698322a6c70efe"
server_key_iv: "c8bec158cabc6e27342280b77899cc30"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.668411 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:24.685281 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:24.688616 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:24.688839 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:24.696085 9045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39635
I20260929 00:57:24.696149 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39635 every 8 connection(s)
I20260929 00:57:24.697307 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:24.701568 9113 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:57:24.703251 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Bootstrap starting.
I20260929 00:57:24.707087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 9045
I20260929 00:57:24.707564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44027
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39635
--webserver_port=39955
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:24.720217 9113 log.cc:824] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:24.727703 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:57:24.728154 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Bootstrap complete.
I20260929 00:57:24.732945 9113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260929 00:57:24.733402 9113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad75345c4d84781a840405d1212b571, State: Initialized, Role: FOLLOWER
I20260929 00:57:24.733958 9113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260929 00:57:24.734112 9113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:24.734229 9113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:24.734354 9113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:57:24.736349 9113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260929 00:57:24.736706 9113 leader_election.cc:302] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ad75345c4d84781a840405d1212b571; no voters:
I20260929 00:57:24.737076 9113 leader_election.cc:288] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:57:24.737178 9117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:57:24.737363 9117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [term 2 LEADER]: Becoming Leader. State: Replica: 6ad75345c4d84781a840405d1212b571, State: Running, Role: LEADER
I20260929 00:57:24.737727 9117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } }
I20260929 00:57:24.738379 9113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:24.739189 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6ad75345c4d84781a840405d1212b571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } } }
I20260929 00:57:24.739333 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:24.739652 9119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad75345c4d84781a840405d1212b571. Latest consensus state: current_term: 2 leader_uuid: "6ad75345c4d84781a840405d1212b571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad75345c4d84781a840405d1212b571" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39635 } } }
I20260929 00:57:24.739775 9119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:24.740456 9125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:24.743286 9125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:24.743851 9125 catalog_manager.cc:1259] Loaded cluster ID: 82edf90d68594bfebbd40b33c934ced5
I20260929 00:57:24.743928 9125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:24.745605 9125 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:24.746169 9125 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6ad75345c4d84781a840405d1212b571: Loaded TSK: 0
I20260929 00:57:24.747202 9125 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:57:24.853252 9115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:24.853578 9115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:24.853677 9115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:24.857781 9115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:24.857880 9115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:24.857969 9115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:24.863505 9115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39955
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:24.864560 9115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:24.865597 9115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:24.876250 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.876226 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.876231 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:24.876709 9115 server_base.cc:1034] running on GCE node
I20260929 00:57:24.877213 9115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:24.877867 9115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:24.879060 9115 hybrid_clock.cc:648] HybridClock initialized: now 1790643444879026 us; error 50 us; skew 500 ppm
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:24.881868 9115 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 00:57:24.882856 9115 webserver.cc:460] Webserver started at http://127.0.98.1:39955/ using document root <none> and password file <none>
I20260929 00:57:24.883436 9115 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:24.883510 9115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:24.887586 9115 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:24.889779 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.891135 9115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:24.891275 9115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "b66397d3027a4243a44a72c5c3178d90"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-30s6"
server_key: "62f6e2c9d243c780e5b4a669c41a5ae8"
server_key_iv: "5890065bfa8b7d5d590e29f210a9a811"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.891852 9115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:24.908262 9115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:24.911502 9115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:24.911710 9115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:24.912304 9115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:24.913189 9115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:24.913261 9115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.913336 9115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:24.913386 9115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.924311 9115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44027
I20260929 00:57:24.924350 9262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44027 every 8 connection(s)
I20260929 00:57:24.925351 9115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:24.934719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 9115
I20260929 00:57:24.935257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46027
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39635
--webserver_port=36249
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:57:24.939059 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:24.927668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45819 (local address 127.0.98.62:39635)
0929 00:57:24.928172 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:24.928189 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:24.928438 (+ 249us) server_negotiation.cc:407] Connection header received
0929 00:57:24.929983 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:24.929994 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:24.930325 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:24.930602 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:24.931663 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.932641 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.933723 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.934215 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.934357 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.938388 (+ 4031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45819: BlockingRecv error: recv got EOF from 127.0.98.1:45819 (error 108)
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":167,"threads_started":1}
I20260929 00:57:24.939141 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.927785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:45819)
0929 00:57:24.928255 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:24.928318 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:24.929605 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:24.930697 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:24.930704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:24.931051 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:24.931507 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.931520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.932761 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.932763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.933609 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.933618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.934367 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.934370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.934593 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.935221 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:24.935241 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:24.938053 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":146,"threads_started":1}
W20260929 00:57:24.939778 9263 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39635 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39635: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:24.946235 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.940325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:35895)
0929 00:57:24.940521 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:24.940538 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:24.940636 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:24.940957 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:24.940962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:24.941152 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:24.941293 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.941302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.942004 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.942010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.943073 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.943086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.943534 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.943540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.943663 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.944134 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:24.944153 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:24.946069 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260929 00:57:24.946309 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:24.940396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35895 (local address 127.0.98.62:39635)
0929 00:57:24.940500 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:24.940504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:24.940606 (+ 102us) server_negotiation.cc:407] Connection header received
0929 00:57:24.940765 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:24.940768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:24.940818 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:24.940907 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:24.941409 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.941894 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.943198 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.943418 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.943496 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.946199 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35895: BlockingRecv error: recv got EOF from 127.0.98.1:35895 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:24 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:24.951730 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.946940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:42385)
0929 00:57:24.947096 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:24.947111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:24.947186 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:24.947426 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:24.947429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:24.947575 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:24.947639 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.947643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.948338 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.948341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.948847 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.948855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.949289 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.949292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.949385 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.949866 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:24.949876 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:24.951585 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 00:57:24.952036 9263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:57:24.951870 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:24.947021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42385 (local address 127.0.98.62:39635)
0929 00:57:24.947168 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:24.947171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:24.947184 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:57:24.947280 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:24.947283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:24.947328 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:24.947411 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:24.947747 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.948224 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.948964 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.949173 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.949277 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.951728 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42385: BlockingRecv error: recv got EOF from 127.0.98.1:42385 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 00:57:25.049103 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:25.049450 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:25.049515 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:25.053771 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:25.053838 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:25.053900 9267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:25.059165 9267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36249
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:25.060124 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:25.061137 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:25.070509 9273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:25.070515 9275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:25.070575 9272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:25.070590 9267 server_base.cc:1034] running on GCE node
I20260929 00:57:25.071189 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:25.071911 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:25.073071 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790643445073056 us; error 24 us; skew 500 ppm
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643445, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:25.075735 9267 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 00:57:25.076573 9267 webserver.cc:460] Webserver started at http://127.0.98.2:36249/ using document root <none> and password file <none>
I20260929 00:57:25.077116 9267 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:25.077224 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:25.081048 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:25.083141 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.084349 9267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:25.084492 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "76fbbccb606547d0bff3e19eed046330"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-30s6"
server_key: "2d476a722e230d304e97b6a3fbdf0fb7"
server_key_iv: "b92ec4ff0bb45e8dcf7ee1438817e328"
server_key_version: "encryptionkey@0"
I20260929 00:57:25.085040 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:25.096177 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:25.098953 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:25.099150 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:25.099714 9267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:25.100522 9267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:25.100592 9267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.100662 9267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:25.100715 9267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.109773 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46027
I20260929 00:57:25.109807 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46027 every 8 connection(s)
I20260929 00:57:25.110821 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:25.112147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 9267
I20260929 00:57:25.112584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45457
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39635
--webserver_port=44389
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:57:25.122656 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:25.112754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35803 (local address 127.0.98.62:39635)
0929 00:57:25.112889 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.112893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.113854 (+ 961us) server_negotiation.cc:407] Connection header received
0929 00:57:25.115345 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.115349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.115416 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.115523 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.116800 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.117328 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.118553 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.118776 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.118889 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.122530 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35803: BlockingRecv error: recv got EOF from 127.0.98.2:35803 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:57:25.123332 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.113192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:35803)
0929 00:57:25.113684 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.113735 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.115116 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.115707 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.115714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.116161 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.116641 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.116655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.117455 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.117459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.118427 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.118436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.118931 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.118935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.119100 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.119764 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.119780 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.122409 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":151,"threads_started":1}
W20260929 00:57:25.123889 9396 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39635 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39635: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:25.129362 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.124359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:38141)
0929 00:57:25.124476 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.124492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.124571 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.124815 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.124818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.124973 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.125057 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.125062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.125775 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.125779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.126391 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.126400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.126831 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.126834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.126931 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.127414 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.127425 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.129221 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 00:57:25.129457 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:25.124447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38141 (local address 127.0.98.62:39635)
0929 00:57:25.124567 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.124571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.124585 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:57:25.124665 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.124668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.124713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.124801 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.125168 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.125662 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.126505 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.126721 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.126832 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.129346 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38141: BlockingRecv error: recv got EOF from 127.0.98.2:38141 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:25.134790 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.130027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:48069)
0929 00:57:25.130120 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.130136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.130235 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.130477 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.130479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.130623 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.130687 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.130692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.131415 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.131420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.131940 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.131947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.132398 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.132401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.132477 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.132953 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.132964 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.134632 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 00:57:25.134829 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:25.130119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48069 (local address 127.0.98.62:39635)
0929 00:57:25.130283 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.130289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.130303 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:57:25.130336 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.130339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.130383 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.130464 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.130822 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.131309 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.132064 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.132290 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.132396 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.134734 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48069: BlockingRecv error: recv got EOF from 127.0.98.2:48069 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 00:57:25.135103 9396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:57:25.227366 9399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:25.227686 9399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:25.227735 9399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:25.231833 9399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:25.231889 9399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:25.231973 9399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:25.237183 9399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44389
--tserver_master_addrs=127.0.98.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:25.238332 9399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:25.239457 9399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:25.249145 9404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:25.249258 9399 server_base.cc:1034] running on GCE node
W20260929 00:57:25.249130 9407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:25.249162 9405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:25.249780 9399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:25.250483 9399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:25.251641 9399 hybrid_clock.cc:648] HybridClock initialized: now 1790643445251638 us; error 34 us; skew 500 ppm
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643445, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:25.254465 9399 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 00:57:25.255249 9399 webserver.cc:460] Webserver started at http://127.0.98.3:44389/ using document root <none> and password file <none>
I20260929 00:57:25.255759 9399 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:25.255836 9399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:25.259528 9399 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:57:25.261548 9414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.262812 9399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:25.262903 9399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "14cbc2f9d8424f11bccf71a5f6540622"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-30s6"
server_key: "c368ce6ecc5f9d893a2182fc0a8cf6b8"
server_key_iv: "a251bd5838e055a368b3a4c5fdec0893"
server_key_version: "encryptionkey@0"
I20260929 00:57:25.263405 9399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:25.279450 9399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:25.282944 9399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:25.283102 9399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:25.283658 9399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:25.284436 9399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:25.284482 9399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.284523 9399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:25.284539 9399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.294380 9399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45457
I20260929 00:57:25.294399 9527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45457 every 8 connection(s)
I20260929 00:57:25.295557 9399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:25.300957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 9399
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:57:25.307696 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:25.297544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35475 (local address 127.0.98.62:39635)
0929 00:57:25.297707 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.297712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.298749 (+ 1037us) server_negotiation.cc:407] Connection header received
0929 00:57:25.300090 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.300093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.300140 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.300223 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.301486 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.302155 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.303748 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.303974 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.304086 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.307572 (+ 3486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35475: BlockingRecv error: recv got EOF from 127.0.98.3:35475 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:25.308385 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.298050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:35475)
0929 00:57:25.298581 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.298627 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.299875 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.300397 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.300404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.300768 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.301334 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.301349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.302332 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.302335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.303626 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.303636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.304128 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.304131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.304296 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.304902 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.304920 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.307474 (+ 2554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":191,"threads_started":1}
W20260929 00:57:25.308851 9528 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39635 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39635: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260929 00:57:25.310873 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.302029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.0.98.62:39635)
0929 00:57:25.302290 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.302293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.302305 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:57:25.302342 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.302344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.302386 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.302462 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.303393 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.303903 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.304802 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.305034 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.305147 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.306657 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:25.306671 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:25.306688 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:25.306716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:25.308628 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.309093 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.309099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.309105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.309245 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.309608 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.309612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.309614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.309956 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:25.310164 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.310474 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.310708 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:25.315091 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.309255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:58977)
0929 00:57:25.309343 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.309359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.309435 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.309661 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.309664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.309814 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.309900 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.309905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.310754 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.310758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.311549 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.311559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.312036 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.312040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.312140 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.312619 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.312636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.314989 (+ 2353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 00:57:25.315256 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:25.309351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58977 (local address 127.0.98.62:39635)
0929 00:57:25.309455 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.309461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.309476 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:25.309512 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.309514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.309558 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.309644 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.310011 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.310642 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.311675 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.311910 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.312025 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.315109 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58977: BlockingRecv error: recv got EOF from 127.0.98.3:58977 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:25.320073 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.315623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:55689)
0929 00:57:25.315769 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.315785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.315857 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.316029 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.316031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.316164 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.316232 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.316236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.316905 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.316908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.317414 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.317421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.317887 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.317890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.317945 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.318392 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.318403 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.319984 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 00:57:25.320240 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:25.315715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55689 (local address 127.0.98.62:39635)
0929 00:57:25.315831 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.315834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.315845 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:57:25.315923 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.315925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.315957 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.316014 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.316318 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.316819 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.317530 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.317772 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.317866 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.320097 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55689: BlockingRecv error: recv got EOF from 127.0.98.3:55689 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 00:57:25.320317 9528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:25 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:25.958283 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.952692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:41885)
0929 00:57:25.952976 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.952989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.953073 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.953333 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.953335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.953508 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.953617 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.953622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.954551 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.954554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.955128 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.955137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.955560 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.955563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.955635 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.956090 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.956100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.958098 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
W20260929 00:57:25.958444 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:25.952747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41885 (local address 127.0.98.62:39635)
0929 00:57:25.953011 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.953014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.953047 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:57:25.953165 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.953167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.953211 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.953292 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.953713 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.954448 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.955256 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.955463 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.955537 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.958301 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41885: BlockingRecv error: recv got EOF from 127.0.98.1:41885 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
Sep 29 00:57:26 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:26 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:26.141412 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:26.135682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:60919)
0929 00:57:26.135960 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:26.135975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:26.136102 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:26.136393 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:26.136396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:26.136574 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:26.136690 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:26.136695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.137419 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.137422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:26.137968 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:26.137975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.138448 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.138451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:26.138618 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:26.139168 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:26.139180 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:26.141281 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
W20260929 00:57:26.141530 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:26.135782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60919 (local address 127.0.98.62:39635)
0929 00:57:26.135938 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:26.135941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:26.136071 (+ 130us) server_negotiation.cc:407] Connection header received
0929 00:57:26.136212 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:26.136214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:26.136258 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:26.136327 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:26.136797 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.137316 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.138068 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.138352 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.138432 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:26.141414 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60919: BlockingRecv error: recv got EOF from 127.0.98.2:60919 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 00:57:26 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:26 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:26.326603 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:26.320941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:54251)
0929 00:57:26.321220 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:26.321240 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:26.321380 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:26.321652 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:26.321655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:26.321868 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:26.321970 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:26.321975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.322799 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.322804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:26.323403 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:26.323411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.323881 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.323885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:26.323982 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:26.324515 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:26.324527 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:26.326420 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
W20260929 00:57:26.326640 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:26.321019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54251 (local address 127.0.98.62:39635)
0929 00:57:26.321172 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:26.321177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:26.321329 (+ 152us) server_negotiation.cc:407] Connection header received
0929 00:57:26.321501 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:26.321503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:26.321547 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:26.321631 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:26.322086 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.322685 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.323527 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.323764 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.323875 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:26.326539 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54251: BlockingRecv error: recv got EOF from 127.0.98.3:54251 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 00:57:26 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:26 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:26.965183 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:26.959235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:34809)
0929 00:57:26.959504 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:26.959517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:26.959608 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:26.959871 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:26.959873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:26.960041 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:26.960135 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:26.960139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.960933 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.960935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:26.961469 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:26.961476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.961937 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.961940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:26.961997 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:26.962574 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:26.962586 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:26.964961 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
W20260929 00:57:26.965294 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:26.959304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34809 (local address 127.0.98.62:39635)
0929 00:57:26.959582 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:26.959586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:26.959601 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:26.959701 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:26.959704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:26.959752 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:26.959840 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:26.960239 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.960829 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.961598 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.961839 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.961928 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:26.965133 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34809: BlockingRecv error: recv got EOF from 127.0.98.1:34809 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":63,"threads_started":1}
Sep 29 00:57:27 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:27 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:27.148222 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:27.142405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:33393)
0929 00:57:27.142735 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:27.142753 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:27.142872 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:27.143167 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:27.143175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:27.143431 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:27.143600 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.143611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.144391 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.144393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.144981 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.144988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.145396 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.145399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.145511 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.146072 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:27.146088 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:27.148074 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
W20260929 00:57:27.148303 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:27.142591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33393 (local address 127.0.98.62:39635)
0929 00:57:27.142728 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:27.142733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:27.142835 (+ 102us) server_negotiation.cc:407] Connection header received
0929 00:57:27.142989 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:27.142991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:27.143038 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:27.143108 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:27.143734 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.144285 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.145082 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.145294 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.145368 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.148210 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33393: BlockingRecv error: recv got EOF from 127.0.98.2:33393 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 00:57:27 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:27 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:27.334416 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:27.327676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:53481)
0929 00:57:27.328046 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:27.328062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:27.328180 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:27.328561 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:27.328563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:27.328807 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:27.328936 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.328953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.330241 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.330245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.330985 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.330994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.331500 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.331503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.331570 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.332096 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:27.332107 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:27.334277 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
W20260929 00:57:27.334681 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:27.327922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53481 (local address 127.0.98.62:39635)
0929 00:57:27.328078 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:27.328082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:27.328127 (+ 45us) server_negotiation.cc:407] Connection header received
0929 00:57:27.328356 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:27.328359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:27.328411 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:27.328490 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:27.329141 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.330056 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.331103 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.331335 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.331422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.334411 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53481: BlockingRecv error: recv got EOF from 127.0.98.3:53481 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 00:57:27 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:27 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:27.977798 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:27.966221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:43175)
0929 00:57:27.966513 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:27.966527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:27.966639 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:27.966896 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:27.966899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:27.967115 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:27.967232 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.967243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.968005 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.968010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.968585 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.968593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.969109 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.969112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.969221 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.975798 (+ 6577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:27.975812 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:27.977677 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
W20260929 00:57:27.977974 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:27.966323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43175 (local address 127.0.98.62:39635)
0929 00:57:27.966549 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:27.966552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:27.966584 (+ 32us) server_negotiation.cc:407] Connection header received
0929 00:57:27.966723 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:27.966725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:27.966768 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:27.966874 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:27.967346 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.967891 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.968711 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.968984 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.969085 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.977831 (+ 8746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43175: BlockingRecv error: recv got EOF from 127.0.98.1:43175 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":57,"threads_started":1}
Sep 29 00:57:28 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:28 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:28.154783 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.149100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:36759)
0929 00:57:28.149426 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.149442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.149601 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.149881 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.149884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.150095 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:28.150248 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.150255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.151005 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.151011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.151610 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.151618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.152062 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.152066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.152143 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.152710 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:28.152732 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:28.154652 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":91,"threads_started":1}
W20260929 00:57:28.154929 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:28.149224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36759 (local address 127.0.98.62:39635)
0929 00:57:28.149393 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.149397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.149535 (+ 138us) server_negotiation.cc:407] Connection header received
0929 00:57:28.149706 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.149709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.149762 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.149858 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:28.150370 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.150892 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.151733 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.151952 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.152059 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.154779 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36759: BlockingRecv error: recv got EOF from 127.0.98.2:36759 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 00:57:28 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:28 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:28.340619 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.335220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:59589)
0929 00:57:28.335534 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.335548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.335651 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.335886 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.335888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.336072 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:28.336173 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.336179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.336905 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.336908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.337514 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.337523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.337952 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.337955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.338022 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.338555 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:28.338567 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:28.340503 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
W20260929 00:57:28.340763 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:28.335306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59589 (local address 127.0.98.62:39635)
0929 00:57:28.335434 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.335438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.335617 (+ 179us) server_negotiation.cc:407] Connection header received
0929 00:57:28.335730 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.335733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.335776 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.335843 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:28.336277 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.336777 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.337620 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.337828 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.337902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.340643 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59589: BlockingRecv error: recv got EOF from 127.0.98.3:59589 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 00:57:28 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:28 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:28.984004 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.978703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:43887)
0929 00:57:28.978976 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.978990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.979080 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.979357 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.979359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.979528 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:28.979622 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.979627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.980398 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.980401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.980960 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.980969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.981411 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.981414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.981473 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.982072 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:28.982084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:28.983854 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
W20260929 00:57:28.984203 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:28.978757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43887 (local address 127.0.98.62:39635)
0929 00:57:28.979009 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.979012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.979044 (+ 32us) server_negotiation.cc:407] Connection header received
0929 00:57:28.979166 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.979169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.979213 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.979294 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:28.979720 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.980268 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.981079 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.981293 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.981367 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.984033 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43887: BlockingRecv error: recv got EOF from 127.0.98.1:43887 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
Sep 29 00:57:29 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:29 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:29.161156 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:29.155569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:42375)
0929 00:57:29.155827 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:29.155842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:29.155951 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:29.156195 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:29.156197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:29.156372 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:29.156467 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:29.156472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.157229 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.157232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:29.157804 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:29.157812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.158304 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.158307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:29.158378 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:29.158951 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:29.158963 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:29.161027 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
W20260929 00:57:29.161269 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:29.155697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42375 (local address 127.0.98.62:39635)
0929 00:57:29.155863 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:29.155867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:29.155902 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:57:29.156037 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:29.156039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:29.156082 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:29.156152 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:29.156587 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.157094 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.157918 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.158139 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.158304 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:29.161158 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42375: BlockingRecv error: recv got EOF from 127.0.98.2:42375 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 00:57:29 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:29 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:29.347451 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:29.341425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:54501)
0929 00:57:29.341705 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:29.341718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:29.341819 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:29.342060 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:29.342062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:29.342265 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:29.342359 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:29.342365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.343215 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.343217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:29.343774 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:29.343781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.344224 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.344227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:29.344288 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:29.344878 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:29.344890 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:29.347330 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
W20260929 00:57:29.347656 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:29.341572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54501 (local address 127.0.98.62:39635)
0929 00:57:29.341727 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:29.341731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:29.341782 (+ 51us) server_negotiation.cc:407] Connection header received
0929 00:57:29.341901 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:29.341903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:29.341948 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:29.342021 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:29.342523 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.343096 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.343893 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.344117 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.344209 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:29.347491 (+ 3282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54501: BlockingRecv error: recv got EOF from 127.0.98.3:54501 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 00:57:29 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:29 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:29.991145 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:29.984933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:49039)
0929 00:57:29.985221 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:29.985234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:29.985320 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:29.985589 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:29.985592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:29.985763 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:29.985853 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:29.985858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.986738 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.986741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:29.987438 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:29.987448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.988041 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.988044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:29.988140 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:29.988773 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:29.988787 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:29.990969 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
W20260929 00:57:29.991289 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:29.985075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49039 (local address 127.0.98.62:39635)
0929 00:57:29.985333 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:29.985336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:29.985351 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:29.985389 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:29.985391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:29.985461 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:29.985552 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:29.985998 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.986615 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.987573 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.987905 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.988017 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:29.991118 (+ 3101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49039: BlockingRecv error: recv got EOF from 127.0.98.1:49039 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
Sep 29 00:57:30 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:30 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:30.168095 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.162180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:50687)
0929 00:57:30.162516 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.162530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:30.162611 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:30.162865 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:30.162868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:30.163108 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:30.163204 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:30.163209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.164278 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.164283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:30.164877 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:30.164886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.165310 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.165312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:30.165376 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:30.165936 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:30.165960 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:30.167976 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":122,"threads_started":1}
W20260929 00:57:30.168219 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:30.162372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50687 (local address 127.0.98.62:39635)
0929 00:57:30.162606 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:30.162611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:30.162634 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:57:30.162676 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:30.162678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:30.162716 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:30.162813 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:30.163351 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.164149 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.165000 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.165212 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.165293 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:30.168103 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50687: BlockingRecv error: recv got EOF from 127.0.98.2:50687 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 29 00:57:30 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:30 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:30.354092 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.348455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:41391)
0929 00:57:30.348739 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.348753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:30.348848 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:30.349179 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:30.349183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:30.349411 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:30.349547 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:30.349555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.350609 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.350611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:30.351179 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:30.351187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.351646 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.351649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:30.351706 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:30.352196 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:30.352207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:30.353967 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
W20260929 00:57:30.354234 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:30.348543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41391 (local address 127.0.98.62:39635)
0929 00:57:30.348747 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:30.348752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:30.348832 (+ 80us) server_negotiation.cc:407] Connection header received
0929 00:57:30.348949 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:30.348952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:30.349017 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:30.349113 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:30.349679 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.350512 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.351327 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.351542 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.351631 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:30.354098 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41391: BlockingRecv error: recv got EOF from 127.0.98.3:41391 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 00:57:30 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:30 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:30.997189 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.991864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:41355)
0929 00:57:30.992136 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.992150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:30.992246 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:30.992484 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:30.992486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:30.992656 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:30.992750 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:30.992756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.993466 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.993468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:30.994027 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:30.994034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.994471 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.994474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:30.994536 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:30.995116 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:30.995138 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:30.997071 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
W20260929 00:57:30.997466 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:30.992041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41355 (local address 127.0.98.62:39635)
0929 00:57:30.992271 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:30.992274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:30.992286 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:57:30.992319 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:30.992321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:30.992361 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:30.992445 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:30.992849 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.993360 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.994130 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:30.994362 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:30.994447 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:30.997208 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41355: BlockingRecv error: recv got EOF from 127.0.98.1:41355 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
Sep 29 00:57:31 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:31 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:31.174428 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:31.168890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:36057)
0929 00:57:31.169158 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:31.169172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:31.169274 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:31.169501 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:31.169503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:31.169678 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:31.169768 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:31.169773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.170510 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.170512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:31.171050 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:31.171058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.171584 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.171587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:31.171649 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:31.172172 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:31.172183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:31.174266 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
W20260929 00:57:31.174530 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:31.168981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36057 (local address 127.0.98.62:39635)
0929 00:57:31.169167 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:31.169171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:31.169249 (+ 78us) server_negotiation.cc:407] Connection header received
0929 00:57:31.169353 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:31.169355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:31.169398 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:31.169468 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:31.169862 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.170425 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.171221 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.171490 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.171568 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:31.174387 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36057: BlockingRecv error: recv got EOF from 127.0.98.2:36057 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 00:57:31 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:31 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:31.360771 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:31.354952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:36305)
0929 00:57:31.355277 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:31.355292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:31.355407 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:31.355702 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:31.355704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:31.355894 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:31.356003 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:31.356009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.356787 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.356790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:31.357387 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:31.357396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.357894 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.357897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:31.357961 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:31.358544 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:31.358555 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:31.360574 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
W20260929 00:57:31.360810 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:31.355110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36305 (local address 127.0.98.62:39635)
0929 00:57:31.355274 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:31.355278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:31.355437 (+ 159us) server_negotiation.cc:407] Connection header received
0929 00:57:31.355497 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:31.355500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:31.355550 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:31.355657 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:31.356138 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.356672 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.357523 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.357780 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.357871 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:31.360700 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36305: BlockingRecv error: recv got EOF from 127.0.98.3:36305 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 00:57:31 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:31 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:32.004381 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:31.998050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:59047)
0929 00:57:31.998415 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:31.998438 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:31.998551 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:31.998811 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:31.998813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:31.999004 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:31.999108 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:31.999114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.999893 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:31.999897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:32.000551 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:32.000561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.001041 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.001045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:32.001128 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:32.001824 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:32.001837 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:32.004223 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":149,"threads_started":1}
W20260929 00:57:32.004688 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:31.998167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59047 (local address 127.0.98.62:39635)
0929 00:57:31.998441 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:31.998444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:31.998522 (+ 78us) server_negotiation.cc:407] Connection header received
0929 00:57:31.998627 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:31.998629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:31.998673 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:31.998766 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:31.999222 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:31.999777 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.000678 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.000907 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.000991 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:32.004486 (+ 3495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59047: BlockingRecv error: recv got EOF from 127.0.98.1:59047 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
Sep 29 00:57:32 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:32 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:32.181098 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:32.175327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:55763)
0929 00:57:32.175706 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:32.175719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:32.175878 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:32.176155 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:32.176158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:32.176378 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:32.176475 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:32.176481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.177234 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.177239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:32.177792 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:32.177806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.178309 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.178313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:32.178391 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:32.178913 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:32.178924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:32.180913 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":88,"threads_started":1}
W20260929 00:57:32.181181 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:32.175444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55763 (local address 127.0.98.62:39635)
0929 00:57:32.175649 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:32.175653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:32.175798 (+ 145us) server_negotiation.cc:407] Connection header received
0929 00:57:32.175983 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:32.175987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:32.176044 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:32.176133 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:32.176596 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.177122 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.177918 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.178144 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.178316 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:32.181046 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55763: BlockingRecv error: recv got EOF from 127.0.98.2:55763 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 29 00:57:32 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:32 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:32.367408 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:32.361701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:36129)
0929 00:57:32.361996 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:32.362009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:32.362105 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:32.362384 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:32.362387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:32.362586 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:32.362690 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:32.362696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.363435 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.363444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:32.364104 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:32.364113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.364533 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.364536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:32.364618 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:32.365133 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:32.365145 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:32.367251 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
W20260929 00:57:32.367527 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:32.361783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36129 (local address 127.0.98.62:39635)
0929 00:57:32.361920 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:32.361923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:32.362078 (+ 155us) server_negotiation.cc:407] Connection header received
0929 00:57:32.362170 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:32.362172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:32.362232 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:32.362319 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:32.362799 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.363321 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.364210 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:32.364427 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:32.364501 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:32.367418 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36129: BlockingRecv error: recv got EOF from 127.0.98.3:36129 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 00:57:33 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:33 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:33.017153 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:33.005469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:58715)
0929 00:57:33.005768 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:33.005788 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:33.005884 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:33.006174 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:33.006244 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:33.006439 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:33.006541 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:33.006547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.007366 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.007369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:33.007975 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:33.007984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.008472 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.008475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:33.008550 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:33.014870 (+ 6320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:33.014885 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:33.017027 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
W20260929 00:57:33.017416 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:33.005635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58715 (local address 127.0.98.62:39635)
0929 00:57:33.005910 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:33.005913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:33.005927 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:57:33.005968 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:33.005970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:33.006011 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:33.006102 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:33.006662 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.007189 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.008089 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.008344 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.008427 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:33.017235 (+ 8808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58715: BlockingRecv error: recv got EOF from 127.0.98.1:58715 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":125,"threads_started":1}
Sep 29 00:57:33 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:33 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:33.188086 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:33.182041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:56569)
0929 00:57:33.182356 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:33.182370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:33.182466 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:33.182706 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:33.182708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:33.182887 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:33.182980 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:33.182985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.184026 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.184030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:33.184766 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:33.184776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.185194 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.185197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:33.185259 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:33.185962 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:33.185975 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:33.187964 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":97,"threads_started":1}
W20260929 00:57:33.188340 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:33.182103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56569 (local address 127.0.98.62:39635)
0929 00:57:33.182301 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:33.182308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:33.182440 (+ 132us) server_negotiation.cc:407] Connection header received
0929 00:57:33.182553 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:33.182555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:33.182601 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:33.182664 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:33.183079 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.183902 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.184871 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.185089 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.185174 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:33.188210 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56569: BlockingRecv error: recv got EOF from 127.0.98.2:56569 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 00:57:33 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:33 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:33.374078 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:33.368292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:40679)
0929 00:57:33.368549 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:33.368562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:33.368667 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:33.368913 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:33.368915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:33.369090 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:33.369189 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:33.369195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.370134 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.370141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:33.370953 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:33.370966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.371479 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.371483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:33.371567 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:33.372072 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:33.372088 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:33.373950 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
W20260929 00:57:33.374258 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:33.368365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40679 (local address 127.0.98.62:39635)
0929 00:57:33.368500 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:33.368507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:33.368661 (+ 154us) server_negotiation.cc:407] Connection header received
0929 00:57:33.368744 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:33.368746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:33.368803 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:33.368881 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:33.369296 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.369877 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.371076 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:33.371348 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:33.371460 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:33.374076 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40679: BlockingRecv error: recv got EOF from 127.0.98.3:40679 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 00:57:34 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:34 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:34.023892 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:34.018004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:40005)
0929 00:57:34.018381 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:34.018397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:34.018515 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:34.018815 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:34.018818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:34.019071 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:34.019200 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:34.019208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.019986 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.019991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:34.020579 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:34.020588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.021076 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.021079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:34.021177 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:34.021685 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:34.021697 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:34.023721 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":79,"threads_started":1}
W20260929 00:57:34.024049 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:34.018051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40005 (local address 127.0.98.62:39635)
0929 00:57:34.018354 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:34.018363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:34.018483 (+ 120us) server_negotiation.cc:407] Connection header received
0929 00:57:34.018635 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:34.018638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:34.018693 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:34.018793 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:34.019326 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.019874 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.020703 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.020939 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.021073 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:34.023886 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40005: BlockingRecv error: recv got EOF from 127.0.98.1:40005 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
Sep 29 00:57:34 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:34 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:34.194263 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:34.188976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:47205)
0929 00:57:34.189242 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:34.189256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:34.189358 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:34.189576 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:34.189578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:34.189758 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:34.189852 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:34.189856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.190638 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.190641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:34.191200 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:34.191208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.191596 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.191598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:34.191656 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:34.192138 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:34.192149 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:34.194063 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
W20260929 00:57:34.194367 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:34.188997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47205 (local address 127.0.98.62:39635)
0929 00:57:34.189148 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:34.189152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:34.189328 (+ 176us) server_negotiation.cc:407] Connection header received
0929 00:57:34.189438 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:34.189440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:34.189485 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:34.189542 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:34.189947 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.190540 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.191301 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.191505 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.191578 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:34.194252 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47205: BlockingRecv error: recv got EOF from 127.0.98.2:47205 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 00:57:34 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:34 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:34.380576 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:34.374831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:51681)
0929 00:57:34.375094 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:34.375122 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:34.375222 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:34.375449 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:34.375451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:34.375624 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:34.375720 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:34.375725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.376446 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.376449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:34.377398 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:34.377409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.378015 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.378018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:34.378082 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:34.378583 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:34.378594 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:34.380384 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
W20260929 00:57:34.380903 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:34.374945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51681 (local address 127.0.98.62:39635)
0929 00:57:34.375098 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:34.375101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:34.375189 (+ 88us) server_negotiation.cc:407] Connection header received
0929 00:57:34.375297 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:34.375299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:34.375347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:34.375412 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:34.375813 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.376325 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.377540 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:34.377893 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:34.377979 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:34.380545 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51681: BlockingRecv error: recv got EOF from 127.0.98.3:51681 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 00:57:35 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:35 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:35.029947 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:35.024624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:51223)
0929 00:57:35.024891 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:35.024904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:35.024985 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:35.025211 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:35.025213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:35.025385 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:35.025479 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:35.025484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.026285 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.026288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:35.026848 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:35.026856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.027355 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.027358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:35.027427 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:35.027918 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:35.027931 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:35.029774 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
W20260929 00:57:35.030025 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:35.024699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51223 (local address 127.0.98.62:39635)
0929 00:57:35.024927 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:35.024959 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:35.024972 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:57:35.025064 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:35.025066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:35.025108 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:35.025175 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:35.025587 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.026148 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.027024 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.027248 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.027342 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:35.029894 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51223: BlockingRecv error: recv got EOF from 127.0.98.1:51223 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
Sep 29 00:57:35 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:35 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:35.201037 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:35.195286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:39723)
0929 00:57:35.195548 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:35.195586 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:35.195700 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:35.195931 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:35.195933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:35.196188 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:35.196284 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:35.196290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.197145 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.197148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:35.197698 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:35.197706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.198138 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.198141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:35.198270 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:35.198776 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:35.198786 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:35.200868 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
W20260929 00:57:35.201198 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:35.195382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39723 (local address 127.0.98.62:39635)
0929 00:57:35.195575 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:35.195580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:35.195653 (+ 73us) server_negotiation.cc:407] Connection header received
0929 00:57:35.195778 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:35.195780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:35.195821 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:35.195897 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:35.196431 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.197033 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.197814 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.198029 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.198132 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:35.201075 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39723: BlockingRecv error: recv got EOF from 127.0.98.2:39723 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 00:57:35 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:35 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:35.387530 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:35.381480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:55619)
0929 00:57:35.381794 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:35.381812 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:35.381943 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:35.382264 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:35.382267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:35.382488 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:35.382587 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:35.382593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.383392 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.383396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:35.383992 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:35.384000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.384498 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.384502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:35.384599 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:35.385190 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:35.385209 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:35.387335 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":112,"threads_started":1}
W20260929 00:57:35.387609 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:35.381613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55619 (local address 127.0.98.62:39635)
0929 00:57:35.381761 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:35.381769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:35.381902 (+ 133us) server_negotiation.cc:407] Connection header received
0929 00:57:35.382065 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:35.382068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:35.382127 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:35.382239 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:35.382727 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.383267 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.384131 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:35.384380 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:35.384491 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:35.387483 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55619: BlockingRecv error: recv got EOF from 127.0.98.3:55619 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 00:57:36 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:36 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:36.036798 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:36.030780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:35661)
0929 00:57:36.031070 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:36.031084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:36.031178 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:36.031451 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:36.031453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:36.031640 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:36.031743 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:36.031748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.032523 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.032526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:36.033150 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:36.033160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.033591 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.033594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:36.033660 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:36.034296 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:36.034315 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:36.036679 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
W20260929 00:57:36.036967 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:36.030868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35661 (local address 127.0.98.62:39635)
0929 00:57:36.031109 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:36.031112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:36.031128 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:57:36.031266 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:36.031268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:36.031310 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:36.031387 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:36.031848 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.032381 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.033265 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.033484 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.033562 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:36.036820 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35661: BlockingRecv error: recv got EOF from 127.0.98.1:35661 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
Sep 29 00:57:36 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:36 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:36.207835 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:36.202092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:54245)
0929 00:57:36.202401 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:36.202416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:36.202559 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:36.202832 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:36.202835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:36.203056 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:36.203146 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:36.203152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.203922 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.203927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:36.204493 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:36.204501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.204963 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.204966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:36.205060 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:36.205588 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:36.205599 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:36.207718 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
W20260929 00:57:36.207923 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:36.202174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54245 (local address 127.0.98.62:39635)
0929 00:57:36.202374 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:36.202379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:36.202505 (+ 126us) server_negotiation.cc:407] Connection header received
0929 00:57:36.202668 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:36.202671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:36.202724 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:36.202812 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:36.203267 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.203809 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.204615 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.204846 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.204956 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:36.207834 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54245: BlockingRecv error: recv got EOF from 127.0.98.2:54245 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 00:57:36 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:36 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:36.394244 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:36.388472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:51611)
0929 00:57:36.388746 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:36.388768 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:36.388895 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:36.389171 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:36.389174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:36.389384 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:36.389474 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:36.389499 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.390330 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.390333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:36.390890 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:36.390899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.391340 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.391343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:36.391403 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:36.391962 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:36.391973 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:36.394058 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
W20260929 00:57:36.394326 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:36.388545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51611 (local address 127.0.98.62:39635)
0929 00:57:36.388704 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:36.388708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:36.388842 (+ 134us) server_negotiation.cc:407] Connection header received
0929 00:57:36.389001 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:36.389004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:36.389054 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:36.389150 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:36.389629 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.390157 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.391004 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:36.391241 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:36.391324 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:36.394228 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51611: BlockingRecv error: recv got EOF from 127.0.98.3:51611 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 00:57:37 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:37 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:37.043259 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:37.037600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:53875)
0929 00:57:37.037887 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:37.037900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:37.037993 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:37.038397 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:37.038400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:37.038582 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:37.038672 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:37.038677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.039550 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.039553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:37.040120 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:37.040128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.040636 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.040640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:37.040725 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:37.041226 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:37.041237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:37.043141 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
W20260929 00:57:37.043468 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:37.037723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53875 (local address 127.0.98.62:39635)
0929 00:57:37.037946 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:37.037949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:37.037962 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:57:37.038080 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:37.038082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:37.038125 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:37.038240 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:37.038820 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.039412 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.040291 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.040512 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.040620 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:37.043281 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53875: BlockingRecv error: recv got EOF from 127.0.98.1:53875 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":56,"threads_started":1}
Sep 29 00:57:37 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:37 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:37.215008 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:37.208782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:52951)
0929 00:57:37.209052 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:37.209068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:37.209184 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:37.209546 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:37.209548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:37.209811 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:37.209957 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:37.209966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.211096 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.211099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:37.211754 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:37.211764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.212275 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.212278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:37.212347 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:37.212995 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:37.213009 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:37.214889 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
W20260929 00:57:37.215296 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:37.208871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52951 (local address 127.0.98.62:39635)
0929 00:57:37.209044 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:37.209049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:37.209168 (+ 119us) server_negotiation.cc:407] Connection header received
0929 00:57:37.209303 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:37.209306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:37.209353 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:37.209494 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:37.210101 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.210933 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.211899 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.212121 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.212205 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:37.215064 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52951: BlockingRecv error: recv got EOF from 127.0.98.2:52951 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 00:57:37 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:37 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:37.401309 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:37.395198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:50595)
0929 00:57:37.395530 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:37.395545 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:37.395692 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:37.396015 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:37.396018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:37.396281 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:37.396389 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:37.396395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.397174 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.397177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:37.397879 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:37.397889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.398430 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.398434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:37.398519 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:37.399114 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:37.399129 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:37.401088 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":89,"threads_started":1}
W20260929 00:57:37.401553 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:37.395294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50595 (local address 127.0.98.62:39635)
0929 00:57:37.395494 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:37.395499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:37.395636 (+ 137us) server_negotiation.cc:407] Connection header received
0929 00:57:37.395845 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:37.395848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:37.395904 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:37.395992 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:37.396523 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.397055 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.398002 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:37.398309 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:37.398416 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:37.401265 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50595: BlockingRecv error: recv got EOF from 127.0.98.3:50595 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 29 00:57:38 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:38 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:38.054814 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:38.044076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:47409)
0929 00:57:38.044375 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:38.044388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:38.044480 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:38.044751 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:38.044753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:38.044956 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:38.045057 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:38.045062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.045868 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.045871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:38.046572 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:38.046584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.047049 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.047052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:38.047116 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:38.052381 (+ 5265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:38.052396 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:38.054640 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":122,"threads_started":1}
W20260929 00:57:38.054948 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:38.044237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47409 (local address 127.0.98.62:39635)
0929 00:57:38.044487 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:38.044491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:38.044505 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:57:38.044544 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:38.044546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:38.044587 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:38.044675 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:38.045186 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.045736 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.046707 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.046936 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.047021 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:38.054829 (+ 7808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47409: BlockingRecv error: recv got EOF from 127.0.98.1:47409 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
Sep 29 00:57:38 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:38 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:38.221755 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:38.215902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:42413)
0929 00:57:38.216282 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:38.216297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:38.216410 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:38.216629 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:38.216630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:38.216819 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:38.216916 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:38.216922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.217687 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.217690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:38.218405 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:38.218415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.218930 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.218934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:38.219026 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:38.219606 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:38.219621 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:38.221640 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":129,"threads_started":1}
W20260929 00:57:38.221963 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:38.216124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42413 (local address 127.0.98.62:39635)
0929 00:57:38.216281 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:38.216284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:38.216356 (+ 72us) server_negotiation.cc:407] Connection header received
0929 00:57:38.216493 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:38.216495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:38.216537 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:38.216614 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:38.217030 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.217554 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.218539 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.218756 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.218929 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:38.221778 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42413: BlockingRecv error: recv got EOF from 127.0.98.2:42413 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 00:57:38 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:38 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:38.407434 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:38.402103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:48675)
0929 00:57:38.402426 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:38.402440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:38.402547 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:38.402806 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:38.402808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:38.402997 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:38.403094 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:38.403100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.403811 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.403814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:38.404375 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:38.404411 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.404817 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.404819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:38.404889 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:38.405393 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:38.405405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:38.407285 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
W20260929 00:57:38.407596 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:38.402301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48675 (local address 127.0.98.62:39635)
0929 00:57:38.402459 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:38.402462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:38.402500 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:57:38.402615 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:38.402617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:38.402662 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:38.402732 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:38.403205 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.403716 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.404511 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:38.404720 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:38.404794 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:38.407484 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48675: BlockingRecv error: recv got EOF from 127.0.98.3:48675 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 00:57:39 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:39 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:39.061445 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:39.055742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:48749)
0929 00:57:39.056022 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:39.056035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:39.056150 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:39.056496 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:39.056500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:39.056741 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:39.056904 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:39.056910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.057693 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.057696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:39.058273 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:39.058281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.058722 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.058725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:39.058783 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:39.059311 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:39.059323 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:39.061323 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
W20260929 00:57:39.061625 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:39.055893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48749 (local address 127.0.98.62:39635)
0929 00:57:39.056234 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:39.056241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:39.056256 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:39.056294 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:39.056297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:39.056349 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:39.056437 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:39.057038 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.057592 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.058402 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.058622 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.058708 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:39.061454 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48749: BlockingRecv error: recv got EOF from 127.0.98.1:48749 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":94,"threads_started":1}
Sep 29 00:57:39 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:39 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:39.228402 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:39.222604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:54713)
0929 00:57:39.222979 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:39.222996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:39.223149 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:39.223434 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:39.223437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:39.223655 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:39.223759 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:39.223764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.224553 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.224558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:39.225160 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:39.225170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.225664 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.225699 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:39.225796 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:39.226379 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:39.226391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:39.228244 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":86,"threads_started":1}
W20260929 00:57:39.228480 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:39.222754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54713 (local address 127.0.98.62:39635)
0929 00:57:39.222947 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:39.222951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:39.223084 (+ 133us) server_negotiation.cc:407] Connection header received
0929 00:57:39.223257 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:39.223261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:39.223315 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:39.223410 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:39.223883 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.224437 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.225293 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.225535 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.225646 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:39.228386 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54713: BlockingRecv error: recv got EOF from 127.0.98.2:54713 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 00:57:39 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:39 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:39.413929 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:39.408374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.3:34559)
0929 00:57:39.408726 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:39.408739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:39.408872 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:39.409236 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:39.409241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:39.409483 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:39.409617 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:39.409628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.410420 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.410423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:39.410978 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:39.410985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.411404 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.411406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:39.411466 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:39.412021 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:39.412033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:39.413791 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":92,"threads_started":1}
W20260929 00:57:39.414131 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:39.408434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34559 (local address 127.0.98.62:39635)
0929 00:57:39.408644 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:39.408649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:39.408921 (+ 272us) server_negotiation.cc:407] Connection header received
0929 00:57:39.408972 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:39.408977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:39.409038 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:39.409160 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:39.409756 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.410317 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.411081 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:39.411306 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:39.411385 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:39.413958 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34559: BlockingRecv error: recv got EOF from 127.0.98.3:34559 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 29 00:57:40 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:40 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:40.068467 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.062529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.1:41047)
0929 00:57:40.062801 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:40.062814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:40.062902 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:40.063196 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:40.063198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:40.063405 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:40.063502 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:40.063514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.064324 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.064326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:40.064908 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:40.064923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.065444 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.065447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:40.065515 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.066057 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:40.066069 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:40.068348 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
W20260929 00:57:40.068671 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:40.062655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41047 (local address 127.0.98.62:39635)
0929 00:57:40.062918 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:40.062921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:40.062936 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:40.062983 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:40.062985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:40.063027 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:40.063128 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:40.063640 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.064208 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.065045 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.065308 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.065406 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.068516 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41047: BlockingRecv error: recv got EOF from 127.0.98.1:41047 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
Sep 29 00:57:40 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:57:40 dist-test-slave-30s6 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:57:40.235109 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.229497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39635 (local address 127.0.98.2:35173)
0929 00:57:40.229781 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:40.229799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:40.229932 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:40.230235 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:40.230238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:40.230445 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:40.230534 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:40.230539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.231323 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.231326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:40.231883 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:40.231891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.232349 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.232353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:40.232435 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.233010 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:40.233021 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:40.234917 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39635: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
W20260929 00:57:40.235139 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:40.229562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35173 (local address 127.0.98.62:39635)
0929 00:57:40.229749 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:40.229754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:40.229882 (+ 128us) server_negotiation.cc:407] Connection header received
0929 00:57:40.230042 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:40.230044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:40.230095 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:40.230184 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:40.230651 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.231189 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.232007 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.232230 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.232333 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.235040 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35173: BlockingRecv error: recv got EOF from 127.0.98.2:35173 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 00:57:40.302640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 9115
I20260929 00:57:40.308254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 9267
I20260929 00:57:40.313676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 9399
I20260929 00:57:40.318826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 9045
2026-09-29T00:57:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18024 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T00:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:40Z Disabled control of system clock
I20260929 00:57:40.348599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35919
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 00:57:40.458096 9595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:40.458419 9595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:40.458454 9595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:40.462451 9595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:40.462507 9595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:40.462522 9595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:40.462539 9595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:40.462586 9595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:40.467370 9595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:40.468410 9595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:40.469760 9595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:40.478247 9600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:40.478255 9601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:40.478288 9595 server_base.cc:1034] running on GCE node
W20260929 00:57:40.478255 9603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:40.479096 9595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:40.480054 9595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:40.481202 9595 hybrid_clock.cc:648] HybridClock initialized: now 1790643460481185 us; error 52 us; skew 500 ppm
I20260929 00:57:40.482971 9595 webserver.cc:460] Webserver started at http://127.0.98.62:36621/ using document root <none> and password file <none>
I20260929 00:57:40.483567 9595 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:40.483628 9595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:40.483855 9595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:40.485868 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "9629de8a3a494e72a078dc501bb432ad"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "d837b45c6667a5f36cac340b2cec6917"
server_key_iv: "765e65490ea96581fed3f04d6e38f3a1"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.486442 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "9629de8a3a494e72a078dc501bb432ad"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "d837b45c6667a5f36cac340b2cec6917"
server_key_iv: "765e65490ea96581fed3f04d6e38f3a1"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.489962 9595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:57:40.492327 9609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:40.493517 9595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:40.493674 9595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "9629de8a3a494e72a078dc501bb432ad"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "d837b45c6667a5f36cac340b2cec6917"
server_key_iv: "765e65490ea96581fed3f04d6e38f3a1"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.493803 9595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:40.505731 9595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:40.506469 9595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:40.506669 9595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:40.513557 9595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35919
I20260929 00:57:40.513566 9661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35919 every 8 connection(s)
I20260929 00:57:40.514603 9595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:40.517710 9662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:40.523185 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad: Bootstrap starting.
I20260929 00:57:40.524251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 9595
I20260929 00:57:40.524402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:40.524683 392 external_mini_cluster.cc:1428] Setting key f21d9e764c4d8fd946861e2106c6433d
I20260929 00:57:40.525979 9662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:40.526793 9662 log.cc:824] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:40.528800 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad: No bootstrap required, opened a new log
I20260929 00:57:40.532215 9662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9629de8a3a494e72a078dc501bb432ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35919 } }
I20260929 00:57:40.532469 9662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:40.532553 9662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9629de8a3a494e72a078dc501bb432ad, State: Initialized, Role: FOLLOWER
I20260929 00:57:40.533075 9662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9629de8a3a494e72a078dc501bb432ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35919 } }
I20260929 00:57:40.533237 9662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:40.533306 9662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:40.533402 9662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:40.534075 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.525773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.98.62:35919)
0929 00:57:40.526532 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:40.526547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:40.526584 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:57:40.527323 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:40.527347 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:40.527766 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:40.528109 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:40.528535 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.529563 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.530521 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.531021 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.531200 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.532065 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:40.532070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:40.532084 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:40.532496 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:40.532567 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:40.532593 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:40.532702 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:40.532896 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 00:57:40.533199 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":159,"threads_started":1}
I20260929 00:57:40.534435 9662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9629de8a3a494e72a078dc501bb432ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35919 } }
I20260929 00:57:40.534777 9662 leader_election.cc:302] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9629de8a3a494e72a078dc501bb432ad; no voters:
I20260929 00:57:40.535079 9662 leader_election.cc:288] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:40.536145 9667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:40.536386 9667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [term 1 LEADER]: Becoming Leader. State: Replica: 9629de8a3a494e72a078dc501bb432ad, State: Running, Role: LEADER
I20260929 00:57:40.536700 9667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9629de8a3a494e72a078dc501bb432ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35919 } }
I20260929 00:57:40.537261 9662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:40.538303 9668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9629de8a3a494e72a078dc501bb432ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9629de8a3a494e72a078dc501bb432ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35919 } } }
I20260929 00:57:40.538434 9668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:40.538714 9669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9629de8a3a494e72a078dc501bb432ad. Latest consensus state: current_term: 1 leader_uuid: "9629de8a3a494e72a078dc501bb432ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9629de8a3a494e72a078dc501bb432ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35919 } } }
I20260929 00:57:40.538800 9676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:40.538892 9669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:40.541587 9676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:40.547722 9676 catalog_manager.cc:1347] Generated new cluster ID: d0026cb118684b49b940b0eb32ed3a85
I20260929 00:57:40.547860 9676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:40.565544 9676 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:40.566521 9676 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:40.575600 9676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9629de8a3a494e72a078dc501bb432ad: Generated new TSK 0
I20260929 00:57:40.576324 9676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:40.580883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35919
--builtin_ntp_servers=127.0.98.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 00:57:40.696113 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:40.696408 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:40.696446 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:40.700767 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:40.700840 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:40.700897 9686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:40.705683 9686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:40.706907 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:40.708252 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:40.717820 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:40.717820 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:40.717820 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:40.718122 9686 server_base.cc:1034] running on GCE node
I20260929 00:57:40.718659 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:40.719378 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:40.720569 9686 hybrid_clock.cc:648] HybridClock initialized: now 1790643460720525 us; error 55 us; skew 500 ppm
I20260929 00:57:40.722604 9686 webserver.cc:460] Webserver started at http://127.0.98.1:41195/ using document root <none> and password file <none>
I20260929 00:57:40.723210 9686 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:40.723305 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:40.723543 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:40.725633 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "cdcdc081ab39457093e6b9348c0fbf9c"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "f31353ccf08695ee4e4efc177d28b329"
server_key_iv: "3572a3b6b527498f51328bd5c6e9ab05"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.726249 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cdcdc081ab39457093e6b9348c0fbf9c"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "f31353ccf08695ee4e4efc177d28b329"
server_key_iv: "3572a3b6b527498f51328bd5c6e9ab05"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.729893 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:40.732569 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:40.733810 9686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 00:57:40.733956 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "cdcdc081ab39457093e6b9348c0fbf9c"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "f31353ccf08695ee4e4efc177d28b329"
server_key_iv: "3572a3b6b527498f51328bd5c6e9ab05"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.734084 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:40.759294 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:40.760021 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:40.760236 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:40.760844 9686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:40.761835 9686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:40.761911 9686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:40.761986 9686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:40.762029 9686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:40.770691 9686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45133
I20260929 00:57:40.770722 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45133 every 8 connection(s)
I20260929 00:57:40.771605 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:40.777418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 9686
I20260929 00:57:40.777541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:40.777834 392 external_mini_cluster.cc:1428] Setting key d93979e6daacbfc46464d63d57029903
I20260929 00:57:40.779783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35919
--builtin_ntp_servers=127.0.98.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 00:57:40.781373 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.773399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60443 (local address 127.0.98.62:35919)
0929 00:57:40.773563 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:40.773567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:40.774707 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 00:57:40.776062 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:40.776065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:40.776110 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:40.776191 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:40.777081 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.777910 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.779326 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.779565 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.779679 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.780886 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:40.780889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:40.780891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:40.780902 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:40.780919 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:40.780927 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:40.781003 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:40.781174 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 00:57:40.781237 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:57:40.782565 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.773844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35919 (local address 127.0.98.1:60443)
0929 00:57:40.774555 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:40.774595 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:40.775865 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:40.776364 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:40.776371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:40.776456 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:40.776958 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:40.776970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.778041 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.778043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:40.779206 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:40.779217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.779734 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.779739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:40.779929 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.780603 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:40.780605 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:40.780644 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:40.780723 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:40.781004 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:40.781015 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:40.781023 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 00:57:40.781156 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 00:57:40.781448 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260929 00:57:40.786810 9814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35919
I20260929 00:57:40.787182 9814 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:40.788058 9814 heartbeater.cc:511] Master 127.0.98.62:35919 requested a full tablet report, sending...
I20260929 00:57:40.790072 9626 ts_manager.cc:194] Registered new tserver with Master: cdcdc081ab39457093e6b9348c0fbf9c (127.0.98.1:45133)
I20260929 00:57:40.791926 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60443
W20260929 00:57:40.897508 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:40.897805 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:40.897845 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:40.902062 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:40.902139 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:40.902221 9817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:40.907094 9817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:40.908175 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:40.909571 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:40.919747 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:40.919731 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:40.919706 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:40.919929 9817 server_base.cc:1034] running on GCE node
I20260929 00:57:40.920445 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:40.921145 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:40.922374 9817 hybrid_clock.cc:648] HybridClock initialized: now 1790643460922360 us; error 26 us; skew 500 ppm
I20260929 00:57:40.924409 9817 webserver.cc:460] Webserver started at http://127.0.98.2:46649/ using document root <none> and password file <none>
I20260929 00:57:40.925079 9817 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:40.925189 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:40.925472 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:40.927783 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb141f56606f4b47abf89b1ccb52fe82"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "f4afe52c850966766151ca12b86595d8"
server_key_iv: "b43abf4f26b5f31e7b41354362a4e67e"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.928402 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb141f56606f4b47abf89b1ccb52fe82"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "f4afe52c850966766151ca12b86595d8"
server_key_iv: "b43abf4f26b5f31e7b41354362a4e67e"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.932448 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 00:57:40.935362 9831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:40.936859 9817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:40.936964 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "fb141f56606f4b47abf89b1ccb52fe82"
format_stamp: "Formatted at 2026-09-29 00:57:40 on dist-test-slave-30s6"
server_key: "f4afe52c850966766151ca12b86595d8"
server_key_iv: "b43abf4f26b5f31e7b41354362a4e67e"
server_key_version: "encryptionkey@0"
I20260929 00:57:40.937063 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:40.952145 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:40.952850 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:40.953007 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:40.953572 9817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:40.954558 9817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:40.954608 9817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:40.954649 9817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:40.954665 9817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:40.963594 9817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39935
I20260929 00:57:40.963691 9944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39935 every 8 connection(s)
I20260929 00:57:40.964618 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:40.966863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 9817
I20260929 00:57:40.966990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:40.967295 392 external_mini_cluster.cc:1428] Setting key de85cf06af234c5c4b7be038924fbff2
I20260929 00:57:40.969498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35919
--builtin_ntp_servers=127.0.98.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 00:57:40.975860 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.966617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41135 (local address 127.0.98.62:35919)
0929 00:57:40.966789 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:40.966795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:40.968492 (+ 1697us) server_negotiation.cc:407] Connection header received
0929 00:57:40.969885 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:40.969888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:40.969929 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:40.969992 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:40.971231 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.971767 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.973638 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.973860 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.973946 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.975355 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:40.975358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:40.975361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:40.975373 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:40.975393 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:40.975401 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:40.975461 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:40.975675 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 00:57:40.975743 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:57:40.977213 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.967102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35919 (local address 127.0.98.2:41135)
0929 00:57:40.967934 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:40.967988 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:40.969652 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:40.970205 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:40.970217 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:40.970340 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:40.971086 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:40.971102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.971904 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.971911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:40.973489 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:40.973504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:40.974039 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:40.974045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:40.974323 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:40.975010 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:40.975013 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:40.975077 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:40.975177 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:40.975492 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:40.975504 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:40.975514 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:57:40.975676 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 00:57:40.976027 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":155,"threads_started":1}
I20260929 00:57:40.978556 9945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35919
I20260929 00:57:40.978813 9945 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:40.979343 9945 heartbeater.cc:511] Master 127.0.98.62:35919 requested a full tablet report, sending...
I20260929 00:57:40.980444 9626 ts_manager.cc:194] Registered new tserver with Master: fb141f56606f4b47abf89b1ccb52fe82 (127.0.98.2:39935)
I20260929 00:57:40.981021 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41135
W20260929 00:57:41.084117 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:41.084422 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:41.084507 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:41.088833 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:41.088923 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:41.089013 9948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:41.093909 9948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:41.095078 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:41.096580 9948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:41.106392 9956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:41.106405 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:41.106395 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:41.106837 9948 server_base.cc:1034] running on GCE node
I20260929 00:57:41.107218 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:41.107838 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:41.109001 9948 hybrid_clock.cc:648] HybridClock initialized: now 1790643461108973 us; error 55 us; skew 500 ppm
I20260929 00:57:41.111033 9948 webserver.cc:460] Webserver started at http://127.0.98.3:43423/ using document root <none> and password file <none>
I20260929 00:57:41.111645 9948 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:41.111713 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:41.111941 9948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:41.114053 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "c9ddef05601347978448d0a7a80bf280"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "973f63b878e7c3f952ca315d32a2254d"
server_key_iv: "d10dfb835a05516cd8073df9b3e9754a"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.114718 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c9ddef05601347978448d0a7a80bf280"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "973f63b878e7c3f952ca315d32a2254d"
server_key_iv: "d10dfb835a05516cd8073df9b3e9754a"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.118419 9948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:41.120965 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.122340 9948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:41.122462 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "c9ddef05601347978448d0a7a80bf280"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "973f63b878e7c3f952ca315d32a2254d"
server_key_iv: "d10dfb835a05516cd8073df9b3e9754a"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.122579 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:41.138309 9948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:41.139067 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:41.139272 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:41.139874 9948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:41.140909 9948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:41.140955 9948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.141029 9948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:41.141067 9948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.149807 9948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37163
I20260929 00:57:41.149828 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37163 every 8 connection(s)
I20260929 00:57:41.150832 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:41.156428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 9948
I20260929 00:57:41.156548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:41.156867 392 external_mini_cluster.cc:1428] Setting key bd15499252cde9d378e01b7718880f67
I20260929 00:57:41.160781 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.152738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58115 (local address 127.0.98.62:35919)
0929 00:57:41.152880 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:41.152884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:41.153997 (+ 1113us) server_negotiation.cc:407] Connection header received
0929 00:57:41.155445 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:41.155448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:41.155502 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:41.155589 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:41.156737 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:41.157400 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:41.158997 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:41.159212 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:41.159295 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:41.160361 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:41.160364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:41.160366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:41.160378 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:41.160396 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:41.160404 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:41.160461 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:41.160603 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 00:57:41.160653 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:41.161648 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.153146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35919 (local address 127.0.98.3:58115)
0929 00:57:41.153844 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:41.153886 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:41.155231 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:41.155773 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:41.155780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:41.155863 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:41.156499 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:41.156523 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:41.157525 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:41.157528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:41.158844 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:41.158855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:41.159365 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:41.159369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:41.159516 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:41.160105 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:41.160106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:41.160146 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:41.160208 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:41.160475 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:41.160481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:41.160486 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:57:41.160581 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 00:57:41.160804 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260929 00:57:41.162758 10076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35919
I20260929 00:57:41.163002 10076 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:41.163514 10076 heartbeater.cc:511] Master 127.0.98.62:35919 requested a full tablet report, sending...
I20260929 00:57:41.164484 9626 ts_manager.cc:194] Registered new tserver with Master: c9ddef05601347978448d0a7a80bf280 (127.0.98.3:37163)
I20260929 00:57:41.165031 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58115
I20260929 00:57:41.170567 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:57:41.173265 9665 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:41.172434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.0.98.62:35919)
0929 00:57:41.172586 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:41.172591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:41.172705 (+ 114us) server_negotiation.cc:407] Connection header received
0929 00:57:41.172786 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:41.172788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:41.172826 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:41.172888 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:41.173102 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44778: BlockingRecv error: recv got EOF from 127.0.0.1:44778 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:57:41.173352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 9686
I20260929 00:57:41.178925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 9817
I20260929 00:57:41.184075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 9948
I20260929 00:57:41.189579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 9595
2026-09-29T00:57:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (869 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T00:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:41Z Disabled control of system clock
I20260929 00:57:41.218381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35681
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 00:57:41.330724 10087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:41.331012 10087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:41.331051 10087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:41.335180 10087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:41.335254 10087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:41.335273 10087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:41.335294 10087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:41.335312 10087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:41.340111 10087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:41.341144 10087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:41.342509 10087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:41.350777 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:41.350773 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:41.350780 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:41.351176 10087 server_base.cc:1034] running on GCE node
I20260929 00:57:41.351718 10087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:41.352748 10087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:41.353909 10087 hybrid_clock.cc:648] HybridClock initialized: now 1790643461353891 us; error 34 us; skew 500 ppm
I20260929 00:57:41.355748 10087 webserver.cc:460] Webserver started at http://127.0.98.62:36095/ using document root <none> and password file <none>
I20260929 00:57:41.356318 10087 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:41.356379 10087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:41.356619 10087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:41.358682 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "71bbbe258c6a467986f09b0cd166e613"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "dc913c766252a5d8be3bfc393f1450fa"
server_key_iv: "2d415e6f51ba378d736cb262d5ce2420"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.359248 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "71bbbe258c6a467986f09b0cd166e613"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "dc913c766252a5d8be3bfc393f1450fa"
server_key_iv: "2d415e6f51ba378d736cb262d5ce2420"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.362730 10087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:41.365029 10101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.366264 10087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:41.366380 10087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "71bbbe258c6a467986f09b0cd166e613"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "dc913c766252a5d8be3bfc393f1450fa"
server_key_iv: "2d415e6f51ba378d736cb262d5ce2420"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.366475 10087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:41.370867 10087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:41.371462 10087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:41.371637 10087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:41.377935 10087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35681
I20260929 00:57:41.377941 10153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35681 every 8 connection(s)
I20260929 00:57:41.378890 10087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:41.381896 10154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:41.383875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 10087
I20260929 00:57:41.384002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:41.384291 392 external_mini_cluster.cc:1428] Setting key f6bb165c48788ff29411d613153e7ad0
I20260929 00:57:41.388199 10154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613: Bootstrap starting.
I20260929 00:57:41.390108 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.385774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.0.98.62:35681)
0929 00:57:41.386471 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:41.386487 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:41.386523 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:57:41.387320 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:41.387337 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:41.387823 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:41.388159 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:41.388923 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:41.388928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:41.388939 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:41.388947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:41.388997 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:41.389012 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:41.389159 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:41.389227 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 00:57:41.389451 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":134,"threads_started":1}
I20260929 00:57:41.390653 10154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:41.391448 10154 log.cc:824] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:41.393253 10154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613: No bootstrap required, opened a new log
I20260929 00:57:41.396689 10154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bbbe258c6a467986f09b0cd166e613" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260929 00:57:41.396946 10154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:41.397011 10154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71bbbe258c6a467986f09b0cd166e613, State: Initialized, Role: FOLLOWER
I20260929 00:57:41.397457 10154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bbbe258c6a467986f09b0cd166e613" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260929 00:57:41.397578 10154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:41.397630 10154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:41.397699 10154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:41.398593 10154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bbbe258c6a467986f09b0cd166e613" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260929 00:57:41.398874 10154 leader_election.cc:302] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71bbbe258c6a467986f09b0cd166e613; no voters:
I20260929 00:57:41.399142 10154 leader_election.cc:288] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:41.399259 10159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:41.399487 10159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [term 1 LEADER]: Becoming Leader. State: Replica: 71bbbe258c6a467986f09b0cd166e613, State: Running, Role: LEADER
I20260929 00:57:41.399858 10159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bbbe258c6a467986f09b0cd166e613" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260929 00:57:41.400337 10154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:41.401465 10161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71bbbe258c6a467986f09b0cd166e613. Latest consensus state: current_term: 1 leader_uuid: "71bbbe258c6a467986f09b0cd166e613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bbbe258c6a467986f09b0cd166e613" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } } }
I20260929 00:57:41.401607 10161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:41.401954 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71bbbe258c6a467986f09b0cd166e613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bbbe258c6a467986f09b0cd166e613" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } } }
I20260929 00:57:41.402045 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:41.402020 10168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:41.404850 10168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:41.410341 10168 catalog_manager.cc:1347] Generated new cluster ID: 36c9b7d424b54eb89814085523b021e7
I20260929 00:57:41.410416 10168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:41.421177 10168 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:41.421530 10168 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:41.431259 10168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71bbbe258c6a467986f09b0cd166e613: Generated new TSK 0
I20260929 00:57:41.431984 10168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:41.438010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--builtin_ntp_servers=127.0.98.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 00:57:41.548883 10178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:41.549136 10178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:41.549176 10178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:41.553457 10178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:41.553515 10178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:41.553613 10178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:41.558595 10178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:41.559676 10178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:41.561045 10178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:41.570484 10186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:41.570489 10184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:41.570484 10183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:41.570729 10178 server_base.cc:1034] running on GCE node
I20260929 00:57:41.571221 10178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:41.571835 10178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:41.573004 10178 hybrid_clock.cc:648] HybridClock initialized: now 1790643461572970 us; error 45 us; skew 500 ppm
I20260929 00:57:41.574868 10178 webserver.cc:460] Webserver started at http://127.0.98.1:40653/ using document root <none> and password file <none>
I20260929 00:57:41.575438 10178 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:41.575497 10178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:41.575724 10178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:41.577739 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "54056b62add94bf5aea281fbc9997cef"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "d380157cae1e2afa92417320a92cde8a"
server_key_iv: "275653f6f8eed1e2d32531c4fe830ed6"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.578342 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "54056b62add94bf5aea281fbc9997cef"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "d380157cae1e2afa92417320a92cde8a"
server_key_iv: "275653f6f8eed1e2d32531c4fe830ed6"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.581894 10178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:41.584263 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.585515 10178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:57:41.585676 10178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "54056b62add94bf5aea281fbc9997cef"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "d380157cae1e2afa92417320a92cde8a"
server_key_iv: "275653f6f8eed1e2d32531c4fe830ed6"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.585793 10178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:41.590408 10178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:41.590991 10178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:41.591169 10178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:41.591703 10178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:41.592567 10178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:41.592646 10178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.592707 10178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:41.592758 10178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.601378 10178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40965
I20260929 00:57:41.601401 10305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40965 every 8 connection(s)
I20260929 00:57:41.602394 10178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:41.604020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 10178
I20260929 00:57:41.604180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:41.604470 392 external_mini_cluster.cc:1428] Setting key f9aa3f56843400d0b86b590a8306f4a0
I20260929 00:57:41.606389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--builtin_ntp_servers=127.0.98.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 00:57:41.608740 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.604236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41659 (local address 127.0.98.62:35681)
0929 00:57:41.604365 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:41.604369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:41.605611 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 00:57:41.606936 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:41.606943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:41.606990 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:41.607082 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:41.608236 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:41.608242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:41.608246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:41.608264 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:41.608290 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:41.608304 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:41.608404 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:41.608532 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 00:57:41.608578 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:57:41.609586 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.604740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:41659)
0929 00:57:41.605454 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:41.605513 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:41.606734 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:41.607234 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:41.607240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:41.607325 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:41.607978 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:41.607980 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:41.608023 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:41.608089 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:41.608393 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:41.608400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:41.608403 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 00:57:41.608552 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 00:57:41.608735 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":128,"threads_started":1}
I20260929 00:57:41.610874 10306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260929 00:57:41.611122 10306 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:41.611524 10306 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260929 00:57:41.613226 10118 ts_manager.cc:194] Registered new tserver with Master: 54056b62add94bf5aea281fbc9997cef (127.0.98.1:40965)
W20260929 00:57:41.720745 10309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:41.721031 10309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:41.721072 10309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:41.725199 10309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:41.725270 10309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:41.725327 10309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:41.730037 10309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:41.731135 10309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:41.732580 10309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:41.742358 10315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:41.742501 10309 server_base.cc:1034] running on GCE node
W20260929 00:57:41.742336 10317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:41.742336 10314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:41.743064 10309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:41.743767 10309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:41.744935 10309 hybrid_clock.cc:648] HybridClock initialized: now 1790643461744915 us; error 31 us; skew 500 ppm
I20260929 00:57:41.746881 10309 webserver.cc:460] Webserver started at http://127.0.98.2:34447/ using document root <none> and password file <none>
I20260929 00:57:41.747570 10309 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:41.747645 10309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:41.747879 10309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:41.749950 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "166e0e22372c48008122d276a6fef3ff"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "8dfd489e1d1caaa2c2c3dfb6280265f3"
server_key_iv: "e1459c3a4a1a1af3aca3af6f4bc2238d"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.750577 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "166e0e22372c48008122d276a6fef3ff"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "8dfd489e1d1caaa2c2c3dfb6280265f3"
server_key_iv: "e1459c3a4a1a1af3aca3af6f4bc2238d"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.754472 10309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:57:41.757002 10323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.758354 10309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:41.758483 10309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "166e0e22372c48008122d276a6fef3ff"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "8dfd489e1d1caaa2c2c3dfb6280265f3"
server_key_iv: "e1459c3a4a1a1af3aca3af6f4bc2238d"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.758617 10309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:41.763258 10309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:41.763865 10309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:41.764048 10309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:41.764601 10309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:41.765504 10309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:41.765575 10309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.765671 10309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:41.765722 10309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.775058 10309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40633
I20260929 00:57:41.775084 10436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40633 every 8 connection(s)
I20260929 00:57:41.775908 10309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:41.781553 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.777644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42381 (local address 127.0.98.62:35681)
0929 00:57:41.777807 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:41.777812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:41.778878 (+ 1066us) server_negotiation.cc:407] Connection header received
0929 00:57:41.780066 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:41.780069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:41.780111 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:41.780184 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:41.781231 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:41.781233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:41.781235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:41.781243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:41.781258 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:41.781265 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:41.781331 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:41.781419 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 00:57:41.781453 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:57:41.782471 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.778040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:42381)
0929 00:57:41.778729 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:41.778769 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:41.779893 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:41.780308 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:41.780314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:41.780398 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:41.781006 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:41.781008 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:41.781047 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:41.781111 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:41.781317 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:41.781321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:41.781324 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 00:57:41.781437 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 00:57:41.781652 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":167,"threads_started":1}
I20260929 00:57:41.783154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 10309
I20260929 00:57:41.783277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:41.783579 392 external_mini_cluster.cc:1428] Setting key a7d762b437368088e8e9f59c02284fd9
I20260929 00:57:41.783604 10437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260929 00:57:41.783864 10437 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:41.784308 10437 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260929 00:57:41.785327 10118 ts_manager.cc:194] Registered new tserver with Master: 166e0e22372c48008122d276a6fef3ff (127.0.98.2:40633)
I20260929 00:57:41.785542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--builtin_ntp_servers=127.0.98.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 00:57:41.894892 10440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:41.895186 10440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:41.895267 10440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:41.899276 10440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:41.899358 10440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:41.899447 10440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:41.904548 10440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:41.905678 10440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:41.907132 10440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:41.916769 10445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:41.916770 10448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:41.916805 10440 server_base.cc:1034] running on GCE node
W20260929 00:57:41.916769 10446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:41.917496 10440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:41.918138 10440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:41.919351 10440 hybrid_clock.cc:648] HybridClock initialized: now 1790643461919325 us; error 30 us; skew 500 ppm
I20260929 00:57:41.921252 10440 webserver.cc:460] Webserver started at http://127.0.98.3:35871/ using document root <none> and password file <none>
I20260929 00:57:41.921849 10440 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:41.921943 10440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:41.922232 10440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:41.924292 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "187083181bdf4d6f94bfa5535af79eb5"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "37d66eedfa88d89ea4bc0076182401b0"
server_key_iv: "67aa830a9bf4596f9c6c7197172292ef"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.924860 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "187083181bdf4d6f94bfa5535af79eb5"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "37d66eedfa88d89ea4bc0076182401b0"
server_key_iv: "67aa830a9bf4596f9c6c7197172292ef"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.928718 10440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:41.931277 10454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.932641 10440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:41.932772 10440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "187083181bdf4d6f94bfa5535af79eb5"
format_stamp: "Formatted at 2026-09-29 00:57:41 on dist-test-slave-30s6"
server_key: "37d66eedfa88d89ea4bc0076182401b0"
server_key_iv: "67aa830a9bf4596f9c6c7197172292ef"
server_key_version: "encryptionkey@0"
I20260929 00:57:41.932897 10440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:41.937463 10440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:41.938076 10440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:41.938294 10440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:41.938859 10440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:41.939785 10440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:41.939855 10440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.939924 10440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:41.939976 10440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:41.948890 10440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40465
I20260929 00:57:41.948909 10567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40465 every 8 connection(s)
I20260929 00:57:41.949842 10440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:41.951448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 10440
I20260929 00:57:41.951573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:41.951915 392 external_mini_cluster.cc:1428] Setting key 1dfc44c7d0a2f2b48e962a5c320e2b9a
I20260929 00:57:41.956322 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.951709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36219 (local address 127.0.98.62:35681)
0929 00:57:41.951885 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:41.951890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:41.953200 (+ 1310us) server_negotiation.cc:407] Connection header received
0929 00:57:41.954672 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:41.954675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:41.954710 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:41.954787 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:41.955934 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:41.955937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:41.955939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:57:41.955950 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:41.955969 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:57:41.955977 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:41.956055 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:41.956184 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 00:57:41.956221 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:57:41.957247 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.952237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:36219)
0929 00:57:41.953032 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:41.953084 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:41.954486 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:41.954948 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:41.954955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:41.955046 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:41.955671 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:57:41.955672 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:41.955715 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:57:41.955783 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:41.956060 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:41.956066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:41.956070 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:57:41.956204 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 00:57:41.956400 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":163,"threads_started":1}
I20260929 00:57:41.958360 10568 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260929 00:57:41.958587 10568 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:41.959002 10568 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260929 00:57:41.959981 10118 ts_manager.cc:194] Registered new tserver with Master: 187083181bdf4d6f94bfa5535af79eb5 (127.0.98.3:40465)
I20260929 00:57:41.965626 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:57:41.968556 10157 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:57:41.967628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.0.98.62:35681)
0929 00:57:41.967791 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:41.967795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:41.967932 (+ 137us) server_negotiation.cc:407] Connection header received
0929 00:57:41.968086 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:41.968092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:41.968137 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:41.968224 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:41.968454 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56874: BlockingRecv error: recv got EOF from 127.0.0.1:56874 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:57:41.968766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 10178
I20260929 00:57:41.974321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 10309
I20260929 00:57:41.979560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 10440
I20260929 00:57:41.984670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 10087
2026-09-29T00:57:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (795 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:42 dist-test-slave-30s6 krb5kdc[10578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:42 dist-test-slave-30s6 krb5kdc[10578](info): set up 2 sockets
Sep 29 00:57:42 dist-test-slave-30s6 krb5kdc[10578](info): commencing operation
krb5kdc: starting...
W20260929 00:57:43.051296 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:43 dist-test-slave-30s6 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:57:43.173750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:43.280540 10591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:43.280848 10591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:43.280933 10591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:43.284907 10591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:43.285001 10591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:43.285037 10591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:43.285085 10591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:43.285122 10591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:43.290295 10591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:43.291338 10591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:43.292565 10591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:43.300998 10599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:43.300972 10597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:43.300990 10596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:43.301476 10591 server_base.cc:1034] running on GCE node
I20260929 00:57:43.302021 10591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:43.303140 10591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:43.304327 10591 hybrid_clock.cc:648] HybridClock initialized: now 1790643463304309 us; error 31 us; skew 500 ppm
Sep 29 00:57:43 dist-test-slave-30s6 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:43.311144 10591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:57:43.312160 10591 webserver.cc:460] Webserver started at http://127.0.98.62:34615/ using document root <none> and password file <none>
I20260929 00:57:43.312747 10591 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:43.312826 10591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:43.313009 10591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:43.315330 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "6628cded7e93428f8f0a2fe00c862f50"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "77f09d860b19eb502c7eb1ddcb0fa891"
server_key_iv: "175ef08735da55464db974237b947b75"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.315927 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "6628cded7e93428f8f0a2fe00c862f50"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "77f09d860b19eb502c7eb1ddcb0fa891"
server_key_iv: "175ef08735da55464db974237b947b75"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.320186 10591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:43.322880 10606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:43.324142 10591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:43.324246 10591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "6628cded7e93428f8f0a2fe00c862f50"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "77f09d860b19eb502c7eb1ddcb0fa891"
server_key_iv: "175ef08735da55464db974237b947b75"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.324332 10591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:43.336028 10591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:43.339008 10591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:43.339183 10591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:43.346961 10591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39771
I20260929 00:57:43.347324 10658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39771 every 8 connection(s)
I20260929 00:57:43.348091 10591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:43.349699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 10591
I20260929 00:57:43.349874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:57:43.350265 392 external_mini_cluster.cc:1428] Setting key 5ddab7ac2133c17a06549bf7e12582bb
I20260929 00:57:43.351267 10659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:43.358614 10659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50: Bootstrap starting.
Sep 29 00:57:43 dist-test-slave-30s6 krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:43.361302 10659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:43.362308 10659 log.cc:824] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:43.364715 10659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50: No bootstrap required, opened a new log
I20260929 00:57:43.367112 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:43.351762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.0.98.62:39771)
0929 00:57:43.352205 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:43.352228 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:43.352274 (+ 46us) server_negotiation.cc:407] Connection header received
0929 00:57:43.353190 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:43.353214 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:43.353663 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:43.354062 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:43.354874 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.356539 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.357442 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.358207 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.358391 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:43.361402 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:43.361409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:43.361427 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:43.361466 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:43.363634 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:43.364084 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:43.364090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:43.364097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:43.364241 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:43.364514 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:43.364517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:43.364519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:43.365061 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:43.365282 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:43.365478 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 00:57:43.365807 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
I20260929 00:57:43.368947 10659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6628cded7e93428f8f0a2fe00c862f50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } }
I20260929 00:57:43.369174 10659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:43.369266 10659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6628cded7e93428f8f0a2fe00c862f50, State: Initialized, Role: FOLLOWER
I20260929 00:57:43.369745 10659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6628cded7e93428f8f0a2fe00c862f50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } }
I20260929 00:57:43.369884 10659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:43.369935 10659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:43.370012 10659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:43.370993 10659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6628cded7e93428f8f0a2fe00c862f50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } }
I20260929 00:57:43.371294 10659 leader_election.cc:302] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6628cded7e93428f8f0a2fe00c862f50; no voters:
I20260929 00:57:43.371585 10659 leader_election.cc:288] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:43.371717 10664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:43.371985 10664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [term 1 LEADER]: Becoming Leader. State: Replica: 6628cded7e93428f8f0a2fe00c862f50, State: Running, Role: LEADER
I20260929 00:57:43.372287 10664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6628cded7e93428f8f0a2fe00c862f50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } }
I20260929 00:57:43.372960 10659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:43.373940 10666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6628cded7e93428f8f0a2fe00c862f50. Latest consensus state: current_term: 1 leader_uuid: "6628cded7e93428f8f0a2fe00c862f50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6628cded7e93428f8f0a2fe00c862f50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } } }
I20260929 00:57:43.374074 10666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:43.374341 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6628cded7e93428f8f0a2fe00c862f50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6628cded7e93428f8f0a2fe00c862f50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } } }
I20260929 00:57:43.374442 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:43.374974 10674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:43.377650 10674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:43.383143 10674 catalog_manager.cc:1347] Generated new cluster ID: 1bd2c57604fd4844b14ec5154e82a045
I20260929 00:57:43.383224 10674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:43.392890 10674 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:43.393806 10674 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:43.408295 10674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50: Generated new TSK 0
I20260929 00:57:43.409039 10674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:57:43.438028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:43.552918 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:43.553179 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:43.553263 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:43.557399 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:43.557477 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:43.557541 10685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:57:43.562867 10685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:43.564016 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:43.564980 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:43.574235 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:43.574273 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:43.574363 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:43.574769 10685 server_base.cc:1034] running on GCE node
I20260929 00:57:43.575121 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:43.575734 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:43.576920 10685 hybrid_clock.cc:648] HybridClock initialized: now 1790643463576906 us; error 34 us; skew 500 ppm
Sep 29 00:57:43 dist-test-slave-30s6 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:43.579650 10685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:57:43.580456 10685 webserver.cc:460] Webserver started at http://127.0.98.1:32981/ using document root <none> and password file <none>
I20260929 00:57:43.580951 10685 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:43.581015 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:43.581175 10685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:43.583209 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "a3416c02a01540e794fba31cfa1d926a"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "15cd700cf7a62bfdab4cd3e935f0a3fb"
server_key_iv: "2498c3d349f5adc35dd9d53613daf2fb"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.583706 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a3416c02a01540e794fba31cfa1d926a"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "15cd700cf7a62bfdab4cd3e935f0a3fb"
server_key_iv: "2498c3d349f5adc35dd9d53613daf2fb"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.587363 10685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:43.589628 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:43.590989 10685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:43.591162 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "a3416c02a01540e794fba31cfa1d926a"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "15cd700cf7a62bfdab4cd3e935f0a3fb"
server_key_iv: "2498c3d349f5adc35dd9d53613daf2fb"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.591264 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:43.613934 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:43.616889 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:43.617110 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:43.617722 10685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:43.618561 10685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:43.618610 10685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:43.618680 10685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:43.618710 10685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:43.627949 10685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44995
I20260929 00:57:43.627969 10813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44995 every 8 connection(s)
I20260929 00:57:43.628832 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:43.634721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 10685
I20260929 00:57:43.634876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:43.635174 392 external_mini_cluster.cc:1428] Setting key 3fe75a26dd8c01d78166f9c31fda89d1
Sep 29 00:57:43 dist-test-slave-30s6 krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:43.644658 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:43.630762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36401 (local address 127.0.98.62:39771)
0929 00:57:43.630901 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:43.630913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:43.632093 (+ 1180us) server_negotiation.cc:407] Connection header received
0929 00:57:43.633373 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:43.633377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:43.633424 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:43.633508 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:43.634793 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.635472 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.636775 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.637006 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.637081 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:43.640543 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:43.640549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:43.640556 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:43.640590 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:43.642481 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:43.642999 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:43.643005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:43.643010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:43.643112 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:43.643450 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:43.643455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:43.643456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:43.643705 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:43.643814 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:43.644307 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 00:57:43.644459 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:57:43.645351 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:43.631331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39771 (local address 127.0.98.1:36401)
0929 00:57:43.631941 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:43.631986 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:43.633156 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:43.633677 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:43.633684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:43.634039 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:43.634640 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:43.634654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.635636 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.635639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:43.636641 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:43.636652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.637157 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.637164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:43.637413 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:43.638253 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:43.638306 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:43.640330 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:43.642605 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:43.642611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:43.642623 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:43.642889 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:43.643213 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:43.643216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:43.643217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:43.643350 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:43.643810 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:43.643816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:43.644057 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 00:57:43.644215 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 00:57:43.644482 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":185,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:57:43.646606 10814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39771
I20260929 00:57:43.646859 10814 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:43.647342 10814 heartbeater.cc:511] Master 127.0.98.62:39771 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 00:57:43.649030 10623 ts_manager.cc:194] Registered new tserver with Master: a3416c02a01540e794fba31cfa1d926a (127.0.98.1:44995)
I20260929 00:57:43.650565 10623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:57:43.665061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:43.775238 10819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:43.775496 10819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:43.775578 10819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:43.779692 10819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:43.779760 10819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:43.779863 10819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:57:43.785239 10819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:43.786360 10819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:43.787392 10819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:43.796909 10825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:43.796864 10827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:43.796873 10824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:43.797137 10819 server_base.cc:1034] running on GCE node
I20260929 00:57:43.797624 10819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:43.798341 10819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:43.799521 10819 hybrid_clock.cc:648] HybridClock initialized: now 1790643463799512 us; error 31 us; skew 500 ppm
Sep 29 00:57:43 dist-test-slave-30s6 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:43.802256 10819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:57:43.803086 10819 webserver.cc:460] Webserver started at http://127.0.98.2:43957/ using document root <none> and password file <none>
I20260929 00:57:43.803637 10819 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:43.803725 10819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:43.803953 10819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:43.806056 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "c2c9b23b65814b3692ee4721b92fb54e"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "5f8cda596cb12da6481a7d7da89fc5d1"
server_key_iv: "9c3258bbbc1463953f0486cc65c1377e"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.806828 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c2c9b23b65814b3692ee4721b92fb54e"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "5f8cda596cb12da6481a7d7da89fc5d1"
server_key_iv: "9c3258bbbc1463953f0486cc65c1377e"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.810585 10819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:43.813019 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:43.814340 10819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:43.814436 10819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "c2c9b23b65814b3692ee4721b92fb54e"
format_stamp: "Formatted at 2026-09-29 00:57:43 on dist-test-slave-30s6"
server_key: "5f8cda596cb12da6481a7d7da89fc5d1"
server_key_iv: "9c3258bbbc1463953f0486cc65c1377e"
server_key_version: "encryptionkey@0"
I20260929 00:57:43.814523 10819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:43.824757 10819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:43.827692 10819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:43.827898 10819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:43.828482 10819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:43.829311 10819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:43.829358 10819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:43.829397 10819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:43.829418 10819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:43.838268 10819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36227
I20260929 00:57:43.838287 10947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36227 every 8 connection(s)
I20260929 00:57:43.839195 10819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:43.841044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 10819
I20260929 00:57:43.841159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:43.841463 392 external_mini_cluster.cc:1428] Setting key 75a6f073469b078c6230575782b5effb
Sep 29 00:57:43 dist-test-slave-30s6 krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:57:43.855152 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:43.841096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39587 (local address 127.0.98.62:39771)
0929 00:57:43.841441 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:43.841482 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:43.842334 (+ 852us) server_negotiation.cc:407] Connection header received
0929 00:57:43.843816 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:43.843819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:43.843869 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:43.843947 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:43.845591 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.846103 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.848066 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.848287 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.848378 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:43.851456 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:43.851459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:43.851461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:43.851486 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:43.852885 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:43.853382 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:43.853385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:43.853386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:43.853463 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:43.853824 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:43.853830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:43.853834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:43.854081 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:43.854213 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:43.854892 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 00:57:43.855023 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260929 00:57:43.856308 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:43.841563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39771 (local address 127.0.98.2:39587)
0929 00:57:43.842141 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:43.842228 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:43.843600 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:43.844132 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:43.844140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:43.844675 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:43.845423 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:43.845446 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.846264 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.846271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:43.847912 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:43.847926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:43.848464 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:43.848470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:43.848731 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:43.849427 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:43.849450 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:43.851225 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:43.852998 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:43.853004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:43.853017 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:43.853280 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:43.853560 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:43.853562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:43.853563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:43.853699 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:43.854221 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:43.854230 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:43.854597 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 00:57:43.854820 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 00:57:43.855237 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":144,"threads_started":1}
I20260929 00:57:43.857564 10948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39771
I20260929 00:57:43.857825 10948 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:43.858428 10948 heartbeater.cc:511] Master 127.0.98.62:39771 requested a full tablet report, sending...
I20260929 00:57:43.859678 10623 ts_manager.cc:194] Registered new tserver with Master: c2c9b23b65814b3692ee4721b92fb54e (127.0.98.2:36227)
I20260929 00:57:43.860329 10623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:57:43.869808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:43.980361 10953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:43.980648 10953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:43.980729 10953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:43.984776 10953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:43.984862 10953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:43.984956 10953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:57:43.990078 10953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:57:43.991134 10953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:43.992158 10953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:44.001778 10961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:44.001773 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:44.001864 10953 server_base.cc:1034] running on GCE node
W20260929 00:57:44.001770 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:44.002573 10953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:44.003172 10953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:44.004354 10953 hybrid_clock.cc:648] HybridClock initialized: now 1790643464004328 us; error 37 us; skew 500 ppm
Sep 29 00:57:44 dist-test-slave-30s6 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:44.007189 10953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:57:44.008061 10953 webserver.cc:460] Webserver started at http://127.0.98.3:36675/ using document root <none> and password file <none>
I20260929 00:57:44.008615 10953 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:44.008702 10953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:44.008934 10953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:44.011040 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "0086d9f7664e410d985ebce1b59bce12"
format_stamp: "Formatted at 2026-09-29 00:57:44 on dist-test-slave-30s6"
server_key: "c3c38ed67ce0b2995d680673cac453dd"
server_key_iv: "0e9b63f2f76b7d563345fa3100be1ad3"
server_key_version: "encryptionkey@0"
I20260929 00:57:44.011610 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0086d9f7664e410d985ebce1b59bce12"
format_stamp: "Formatted at 2026-09-29 00:57:44 on dist-test-slave-30s6"
server_key: "c3c38ed67ce0b2995d680673cac453dd"
server_key_iv: "0e9b63f2f76b7d563345fa3100be1ad3"
server_key_version: "encryptionkey@0"
I20260929 00:57:44.015229 10953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:57:44.017606 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:44.018967 10953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:57:44.019099 10953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "0086d9f7664e410d985ebce1b59bce12"
format_stamp: "Formatted at 2026-09-29 00:57:44 on dist-test-slave-30s6"
server_key: "c3c38ed67ce0b2995d680673cac453dd"
server_key_iv: "0e9b63f2f76b7d563345fa3100be1ad3"
server_key_version: "encryptionkey@0"
I20260929 00:57:44.019210 10953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:44.040467 10953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:44.043807 10953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:44.044005 10953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:44.044603 10953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:44.045424 10953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:44.045502 10953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:44.045575 10953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:44.045626 10953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:44.054523 10953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38187
I20260929 00:57:44.054556 11081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38187 every 8 connection(s)
I20260929 00:57:44.055439 10953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:44.056142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 10953
I20260929 00:57:44.056270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:44.056564 392 external_mini_cluster.cc:1428] Setting key e9e9a4fc56ca98b377422c59e0ee79f7
Sep 29 00:57:44 dist-test-slave-30s6 krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:57:44.068849 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.057610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51673 (local address 127.0.98.62:39771)
0929 00:57:44.057746 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:44.057750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:44.058822 (+ 1072us) server_negotiation.cc:407] Connection header received
0929 00:57:44.060179 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:44.060181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:44.060224 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:44.060294 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:44.061364 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.061876 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.063034 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.063241 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.063320 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.065599 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:44.065602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:44.065604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:44.065627 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:44.067175 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:44.067614 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:44.067617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:44.067618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:44.067688 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:44.067971 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:44.067973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:44.067975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:44.068134 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:44.068207 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:44.068637 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 00:57:44.068731 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:57:44.069644 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.058156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39771 (local address 127.0.98.3:51673)
0929 00:57:44.058682 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.058734 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:44.059991 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:44.060423 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:44.060430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:44.060771 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:44.061248 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.061261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.061988 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.061990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.062924 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.062942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.063364 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.063367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.063501 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.064031 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:44.064047 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:44.065452 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:44.067268 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:44.067272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:44.067283 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:44.067530 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:44.067762 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:44.067764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:44.067766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:44.067894 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:44.068217 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:44.068222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:44.068440 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 00:57:44.068592 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 00:57:44.068834 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":156,"threads_started":1}
I20260929 00:57:44.070941 11082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39771
I20260929 00:57:44.071201 11082 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:44.071682 11082 heartbeater.cc:511] Master 127.0.98.62:39771 requested a full tablet report, sending...
I20260929 00:57:44.072692 10622 ts_manager.cc:194] Registered new tserver with Master: 0086d9f7664e410d985ebce1b59bce12 (127.0.98.3:38187)
I20260929 00:57:44.073338 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51673
I20260929 00:57:44.082561 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:57:44.091982 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.084424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.0.98.62:39771)
0929 00:57:44.084588 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:44.084593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:44.084746 (+ 153us) server_negotiation.cc:407] Connection header received
0929 00:57:44.084902 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:44.084905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:44.084954 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:44.085041 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:44.085728 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.086309 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.087151 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.087372 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.087483 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.088843 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:44.088846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:44.088851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:44.088881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:44.090359 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:44.090760 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:44.090765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:44.090767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:44.090847 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:44.091115 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:44.091120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:44.091121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:44.091334 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:44.091484 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:44.091702 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 00:57:44.091814 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:57:44.095160 10622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:44.098310 10622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:44.113210 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.107024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36227 (local address 127.0.0.1:51976)
0929 00:57:44.107687 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.107703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:44.107785 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:44.108481 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:44.108484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:44.108496 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:44.108626 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.108631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.110279 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.110285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.111640 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.111649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.112675 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.112678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.112755 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.112787 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:57:44.112916 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 00:57:44.113040 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":86,"thread_start_us":81,"threads_started":1}
I20260929 00:57:44.113238 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.107768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.0.98.2:36227)
0929 00:57:44.108094 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:44.108104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:44.108122 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:57:44.108184 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:44.108188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:44.108338 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:44.108775 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:44.108933 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.110123 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.111761 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.112548 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.112662 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.112813 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:44.112949 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 00:57:44.113049 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":119,"threads_started":1}
I20260929 00:57:44.113658 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.107669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38187 (local address 127.0.0.1:36256)
0929 00:57:44.108642 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.108653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:44.108771 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:44.109195 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:44.109198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:44.109214 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:44.109327 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.109336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.110320 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.110323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.111918 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.111924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.112926 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.112929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.113026 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.113046 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 00:57:44.113117 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 00:57:44.113171 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":246,"thread_start_us":89,"threads_started":1}
I20260929 00:57:44.113669 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.108106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.0.98.3:38187)
0929 00:57:44.108507 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:44.108520 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:44.108742 (+ 222us) server_negotiation.cc:407] Connection header received
0929 00:57:44.108912 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:44.108916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:44.109065 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:44.109222 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:44.109459 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.110160 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.112031 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.112806 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.112916 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.113058 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:44.113427 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 00:57:44.113533 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":118,"threads_started":1}
I20260929 00:57:44.114847 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.106939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44995 (local address 127.0.0.1:49646)
0929 00:57:44.107387 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.107440 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:44.107577 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:44.108421 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:44.108430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:44.108457 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:44.108609 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.108617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.110320 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.110323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.111895 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.111905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.114477 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.114481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.114574 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.114596 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:57:44.114647 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 00:57:44.114713 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":188,"thread_start_us":67,"threads_started":1}
I20260929 00:57:44.115422 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.107071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.0.98.1:44995)
0929 00:57:44.107868 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:44.107890 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:44.107929 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:57:44.108020 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:44.108027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:44.108241 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:44.108776 (+ 535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:44.108933 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.110120 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.113120 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.114333 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.115108 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.115209 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:44.115248 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 00:57:44.115315 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":114,"threads_started":1}
I20260929 00:57:44.116530 11016 tablet_service.cc:1467] Processing CreateTablet for tablet 050535debaa14bdfaca12909a2be4d31 (DEFAULT_TABLE table=test-table [id=bd69c6ea372841f798a02adf42650668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:44.116530 10882 tablet_service.cc:1467] Processing CreateTablet for tablet 050535debaa14bdfaca12909a2be4d31 (DEFAULT_TABLE table=test-table [id=bd69c6ea372841f798a02adf42650668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:44.117677 10882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050535debaa14bdfaca12909a2be4d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:44.117681 11016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050535debaa14bdfaca12909a2be4d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:44.117610 10748 tablet_service.cc:1467] Processing CreateTablet for tablet 050535debaa14bdfaca12909a2be4d31 (DEFAULT_TABLE table=test-table [id=bd69c6ea372841f798a02adf42650668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:44.118547 10748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050535debaa14bdfaca12909a2be4d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:44.124490 11099 tablet_bootstrap.cc:492] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a: Bootstrap starting.
I20260929 00:57:44.125586 11101 tablet_bootstrap.cc:492] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: Bootstrap starting.
I20260929 00:57:44.126523 11100 tablet_bootstrap.cc:492] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: Bootstrap starting.
I20260929 00:57:44.126783 11099 tablet_bootstrap.cc:654] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:44.127713 11101 tablet_bootstrap.cc:654] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:44.127956 11099 log.cc:824] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:44.128931 11101 log.cc:824] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:44.129164 11100 tablet_bootstrap.cc:654] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:44.130105 11100 log.cc:824] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:44.130398 11099 tablet_bootstrap.cc:492] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a: No bootstrap required, opened a new log
I20260929 00:57:44.130594 11099 ts_tablet_manager.cc:1397] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:57:44.131095 11101 tablet_bootstrap.cc:492] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: No bootstrap required, opened a new log
I20260929 00:57:44.131232 11101 ts_tablet_manager.cc:1397] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:57:44.132042 11100 tablet_bootstrap.cc:492] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: No bootstrap required, opened a new log
I20260929 00:57:44.132200 11100 ts_tablet_manager.cc:1397] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 00:57:44.133864 11101 raft_consensus.cc:348] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.134068 11101 raft_consensus.cc:374] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:44.134099 11101 raft_consensus.cc:729] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0086d9f7664e410d985ebce1b59bce12, State: Initialized, Role: FOLLOWER
I20260929 00:57:44.134565 11101 consensus_queue.cc:260] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.134485 11099 raft_consensus.cc:348] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.134729 11099 raft_consensus.cc:374] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:44.134783 11099 raft_consensus.cc:729] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3416c02a01540e794fba31cfa1d926a, State: Initialized, Role: FOLLOWER
I20260929 00:57:44.135169 11082 heartbeater.cc:503] Master 127.0.98.62:39771 was elected leader, sending a full tablet report...
I20260929 00:57:44.135341 11099 consensus_queue.cc:260] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.135540 11101 ts_tablet_manager.cc:1428] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:44.136299 10814 heartbeater.cc:503] Master 127.0.98.62:39771 was elected leader, sending a full tablet report...
I20260929 00:57:44.136895 11100 raft_consensus.cc:348] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.137212 11100 raft_consensus.cc:374] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:44.137310 11100 raft_consensus.cc:729] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c9b23b65814b3692ee4721b92fb54e, State: Initialized, Role: FOLLOWER
I20260929 00:57:44.137539 11099 ts_tablet_manager.cc:1428] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 00:57:44.137838 11100 consensus_queue.cc:260] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.138485 10948 heartbeater.cc:503] Master 127.0.98.62:39771 was elected leader, sending a full tablet report...
I20260929 00:57:44.138857 11100 ts_tablet_manager.cc:1428] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260929 00:57:44.145017 11107 raft_consensus.cc:482] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:44.145226 11107 raft_consensus.cc:504] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.146270 11107 leader_election.cc:288] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3416c02a01540e794fba31cfa1d926a (127.0.98.1:44995), 0086d9f7664e410d985ebce1b59bce12 (127.0.98.3:38187)
I20260929 00:57:44.151019 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.146893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38693 (local address 127.0.98.3:38187)
0929 00:57:44.147036 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:44.147040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:44.147052 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:57:44.147097 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:44.147103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:44.147151 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:44.147207 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:44.147459 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.148154 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.149660 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.150284 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.150388 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.150452 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:44.150826 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 00:57:44.150894 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:57:44.151019 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.146528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42447 (local address 127.0.98.1:44995)
0929 00:57:44.146664 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:44.146670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:44.146723 (+ 53us) server_negotiation.cc:407] Connection header received
0929 00:57:44.146893 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:44.146896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:44.146947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:44.147022 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:44.147282 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.148068 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.149660 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.150284 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.150388 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.150454 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:44.150826 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 00:57:44.150894 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:57:44.151060 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.146476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44995 (local address 127.0.98.2:42447)
0929 00:57:44.146663 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.146679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:44.146768 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:44.147046 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:44.147048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:44.147060 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:44.147156 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.147160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.148177 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.148179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.149526 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.149537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.150414 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.150418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.150647 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.150723 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 00:57:44.150800 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 00:57:44.150894 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":143}
I20260929 00:57:44.151068 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.146493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38187 (local address 127.0.98.2:38693)
0929 00:57:44.146813 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.146828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:44.146919 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:44.147243 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:44.147247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:44.147264 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:44.147362 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.147366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.148250 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.148253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.149526 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:44.149537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.150414 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.150418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:44.150651 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.150727 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 00:57:44.150800 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 00:57:44.150893 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":111,"thread_start_us":129,"threads_started":1}
I20260929 00:57:44.151750 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050535debaa14bdfaca12909a2be4d31" candidate_uuid: "c2c9b23b65814b3692ee4721b92fb54e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3416c02a01540e794fba31cfa1d926a" is_pre_election: true
I20260929 00:57:44.151818 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050535debaa14bdfaca12909a2be4d31" candidate_uuid: "c2c9b23b65814b3692ee4721b92fb54e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0086d9f7664e410d985ebce1b59bce12" is_pre_election: true
I20260929 00:57:44.152062 10768 raft_consensus.cc:2454] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c9b23b65814b3692ee4721b92fb54e in term 0.
I20260929 00:57:44.152100 11036 raft_consensus.cc:2454] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c9b23b65814b3692ee4721b92fb54e in term 0.
I20260929 00:57:44.152535 10836 leader_election.cc:302] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0086d9f7664e410d985ebce1b59bce12, c2c9b23b65814b3692ee4721b92fb54e; no voters:
I20260929 00:57:44.152778 11107 raft_consensus.cc:2781] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:44.152856 11107 raft_consensus.cc:482] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:44.152899 11107 raft_consensus.cc:3037] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:44.153900 11107 raft_consensus.cc:504] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.154265 11107 leader_election.cc:288] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [CANDIDATE]: Term 1 election: Requested vote from peers a3416c02a01540e794fba31cfa1d926a (127.0.98.1:44995), 0086d9f7664e410d985ebce1b59bce12 (127.0.98.3:38187)
I20260929 00:57:44.154585 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050535debaa14bdfaca12909a2be4d31" candidate_uuid: "c2c9b23b65814b3692ee4721b92fb54e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0086d9f7664e410d985ebce1b59bce12"
I20260929 00:57:44.154585 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050535debaa14bdfaca12909a2be4d31" candidate_uuid: "c2c9b23b65814b3692ee4721b92fb54e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3416c02a01540e794fba31cfa1d926a"
I20260929 00:57:44.154708 11036 raft_consensus.cc:3037] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:44.154724 10768 raft_consensus.cc:3037] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:44.155843 10768 raft_consensus.cc:2454] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c9b23b65814b3692ee4721b92fb54e in term 1.
I20260929 00:57:44.155900 11036 raft_consensus.cc:2454] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c9b23b65814b3692ee4721b92fb54e in term 1.
I20260929 00:57:44.156126 10835 leader_election.cc:302] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3416c02a01540e794fba31cfa1d926a, c2c9b23b65814b3692ee4721b92fb54e; no voters:
I20260929 00:57:44.156302 11107 raft_consensus.cc:2781] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:44.156518 11107 raft_consensus.cc:686] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 1 LEADER]: Becoming Leader. State: Replica: c2c9b23b65814b3692ee4721b92fb54e, State: Running, Role: LEADER
I20260929 00:57:44.156838 11107 consensus_queue.cc:237] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } }
I20260929 00:57:44.160130 10623 catalog_manager.cc:5495] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e reported cstate change: term changed from 0 to 1, leader changed from <none> to c2c9b23b65814b3692ee4721b92fb54e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c2c9b23b65814b3692ee4721b92fb54e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c9b23b65814b3692ee4721b92fb54e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:44.176172 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.172264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.0.98.2:36227)
0929 00:57:44.172396 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:44.172400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:44.172413 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:57:44.172592 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:44.172595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:44.172652 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:44.172744 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:44.172965 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.173451 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.174461 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:44.174720 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:44.174805 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:44.175057 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:44.175515 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:44.175640 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:44.175756 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 00:57:44.175887 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:57:44.184201 10768 raft_consensus.cc:1261] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [term 1 FOLLOWER]: Refusing update from remote peer c2c9b23b65814b3692ee4721b92fb54e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:44.184321 11036 raft_consensus.cc:1261] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 1 FOLLOWER]: Refusing update from remote peer c2c9b23b65814b3692ee4721b92fb54e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:44.185213 11109 consensus_queue.cc:1035] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3416c02a01540e794fba31cfa1d926a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:44.185431 11107 consensus_queue.cc:1035] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0086d9f7664e410d985ebce1b59bce12" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:44.197376 11112 mvcc.cc:205] Tried to move back new op lower bound from 7334475629289340928 to 7334475629189271552. Current Snapshot: MvccSnapshot[applied={T|T < 7334475629289340928 or (T in {7334475629289340928})}]
I20260929 00:57:44.208760 10622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:44.208974 10622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:44.213280 10622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6628cded7e93428f8f0a2fe00c862f50: Sending DeleteTablet for 3 replicas of tablet 050535debaa14bdfaca12909a2be4d31
I20260929 00:57:44.214212 10748 tablet_service.cc:1514] Processing DeleteTablet for tablet 050535debaa14bdfaca12909a2be4d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49646
I20260929 00:57:44.215435 11016 tablet_service.cc:1514] Processing DeleteTablet for tablet 050535debaa14bdfaca12909a2be4d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36256
I20260929 00:57:44.216003 11122 tablet_replica.cc:330] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a: stopping tablet replica
I20260929 00:57:44.216041 10882 tablet_service.cc:1514] Processing DeleteTablet for tablet 050535debaa14bdfaca12909a2be4d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51976
I20260929 00:57:44.216334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 10685
I20260929 00:57:44.216408 11122 raft_consensus.cc:2229] T 050535debaa14bdfaca12909a2be4d31 P a3416c02a01540e794fba31cfa1d926a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:44.217308 11123 tablet_replica.cc:330] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: stopping tablet replica
I20260929 00:57:44.217578 11124 tablet_replica.cc:330] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: stopping tablet replica
I20260929 00:57:44.217689 11123 raft_consensus.cc:2229] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:44.217844 11124 raft_consensus.cc:2229] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:44.218025 11123 raft_consensus.cc:2258] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:44.218236 11124 raft_consensus.cc:2258] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:44.219986 11123 ts_tablet_manager.cc:1905] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:44.223203 11124 ts_tablet_manager.cc:1905] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:44.223939 11123 ts_tablet_manager.cc:1918] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:44.224048 11123 log.cc:1196] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/050535debaa14bdfaca12909a2be4d31
I20260929 00:57:44.224423 11123 ts_tablet_manager.cc:1939] T 050535debaa14bdfaca12909a2be4d31 P 0086d9f7664e410d985ebce1b59bce12: Deleting consensus metadata
I20260929 00:57:44.225724 11124 ts_tablet_manager.cc:1918] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:44.225816 11124 log.cc:1196] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/050535debaa14bdfaca12909a2be4d31
I20260929 00:57:44.225878 10608 catalog_manager.cc:4841] TS 0086d9f7664e410d985ebce1b59bce12 (127.0.98.3:38187): tablet 050535debaa14bdfaca12909a2be4d31 (table test-table [id=bd69c6ea372841f798a02adf42650668]) successfully deleted
I20260929 00:57:44.226125 11124 ts_tablet_manager.cc:1939] T 050535debaa14bdfaca12909a2be4d31 P c2c9b23b65814b3692ee4721b92fb54e: Deleting consensus metadata
I20260929 00:57:44.227183 10608 catalog_manager.cc:4841] TS c2c9b23b65814b3692ee4721b92fb54e (127.0.98.2:36227): tablet 050535debaa14bdfaca12909a2be4d31 (table test-table [id=bd69c6ea372841f798a02adf42650668]) successfully deleted
W20260929 00:57:44.227532 10607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44995 (error 108)
I20260929 00:57:44.228672 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.228317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44995 (local address 127.0.0.1:49662)
0929 00:57:44.228470 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.228556 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 00:57:44.228909 10607 catalog_manager.cc:4584] TS a3416c02a01540e794fba31cfa1d926a (127.0.98.1:44995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 050535debaa14bdfaca12909a2be4d31: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44995: connect: Connection refused (error 111)
I20260929 00:57:44.229393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 10819
I20260929 00:57:44.240302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 10953
I20260929 00:57:44.245836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 10591
2026-09-29T00:57:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2262 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:44 dist-test-slave-30s6 krb5kdc[11128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:44 dist-test-slave-30s6 krb5kdc[11128](info): set up 2 sockets
Sep 29 00:57:44 dist-test-slave-30s6 krb5kdc[11128](info): commencing operation
krb5kdc: starting...
W20260929 00:57:45.275333 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:57:45 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 00:57:45.445199 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIUjItr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/postgres -l logfile start
2026-09-29 00:57:48.093 UTC [11158] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 00:57:48.093 UTC [11158] LOG: listening on IPv4 address "127.0.98.20", port 57733
2026-09-29 00:57:48.096 UTC [11158] LOG: listening on Unix socket "/tmp/.s.PGSQL.57733"
2026-09-29 00:57:48.131 UTC [11161] LOG: database system was shut down at 2026-09-29 00:57:46 UTC
2026-09-29 00:57:48.134 UTC [11158] LOG: database system is ready to accept connections
I20260929 00:57:49.090941 392 mini_postgres.cc:96] Postgres bound to 57733
2026-09-29 00:57:49.097 UTC [11169] FATAL: database "slave" does not exist
127.0.98.20:57733 - accepting connections
I20260929 00:57:49.097705 392 mini_ranger.cc:161] Starting Ranger...
I20260929 00:57:49.112809 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 00:57:49.607919 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 00:57:49.608026 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-admin
2026-09-29 00:57:49,889 [I] DB FLAVOR :POSTGRES
2026-09-29 00:57:49,889 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:57:49,889 [I] Checking connection..
2026-09-29 00:57:49,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 00:57:50,130 [I] Checking connection passed.
2026-09-29 00:57:50,131 [I] --------- Verifying version history table ---------
2026-09-29 00:57:50,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:57:50,388 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:57:50,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:57:50,631 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:57:50,632 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 00:57:50,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 00:57:50.877 UTC [11233] WARNING: there is no transaction in progress
2026-09-29 00:57:50,883 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 00:57:50,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:57:51,134 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 00:57:51,135 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:57:51,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 00:57:51,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 00:57:51,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-30s6.c.gcp-upstream.internal','N') ;"
2026-09-29 00:57:51,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:57:52,102 [I] Table x_portal_user does not exist in database ranger
2026-09-29 00:57:52,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:57:52,346 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 00:57:52,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:57:52,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:57:52,835 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 00:57:52,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 00:57:53.831 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:53.848 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:53.865 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:53.881 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:53.898 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:53.916 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:53.980 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:53.987 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54.002 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54.012 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54.022 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54.038 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54.048 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54.055 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54.062 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54.069 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 00:57:54,802 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 00:57:54,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 00:57:55,055 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 00:57:55,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:57:55,326 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 00:57:55,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:57:55,610 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 00:57:55,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:57:55,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:57:56,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-30s6.c.gcp-upstream.internal';"
2026-09-29 00:57:56,372 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 00:57:56,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 00:57:56,612 [I] DB_PATCHES have already been applied
I20260929 00:57:56.617835 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIUjItr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 00:57:56.617928 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 00:57:56.621476 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:51933
Sep 29, 2026 12:57:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:57:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:57:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 12:57:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 12:57:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 12:57:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 12:57:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 00:57:57.744545 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.647s user 0.000s sys 0.010s
Sep 29, 2026 12:58:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 12:58:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 12:58:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 12:58:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 12:58:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 00:58:19.315439 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 00:58:19 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643499, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 00:58:19.400265 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 00:58:19.409868 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 00:58:20.287040 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 00:58:20.287140 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms
I20260929 00:58:20.287240 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57733
2026-09-29 00:58:20,475 [I] DB FLAVOR :POSTGRES
2026-09-29 00:58:20,476 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:58:20,476 [I] Checking connection
2026-09-29 00:58:20,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 00:58:20,712 [I] connection success
2026-09-29 00:58:20,712 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 00:58:20,712 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 00:58:20,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 00:58:20,968 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 00:58:20,968 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:58:20,968 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 00:58:20,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 00:58:21,213 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 00:58:21.660780 392 mini_ranger_kms.cc:310] Created kms service
I20260929 00:58:21.875828 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 00:58:21.917987 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 00:58:21 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643499, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:58:22.317842 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 00:58:22.318172 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 00:58:22.318231 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 00:58:22.321687 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11521
I20260929 00:58:22.321825 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:47707
Sep 29, 2026 12:58:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:58:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:58:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 12:58:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 12:58:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 12:58:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 00:58:24 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643504, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:24.637140 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.237s user 0.012s sys 0.160s
I20260929 00:58:24.637280 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 00:58:24.637410 392 mini_ranger_kms.cc:401] 127.0.98.20:47707/kms/v1/keys
Sep 29 00:58:25 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643504, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 00:58:25 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643504, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 00:58:27 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:58:27.593663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36473
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36473
--ranger_config_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47707/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:27.718634 11572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:27.718915 11572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:27.718958 11572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:27.723055 11572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:58:27.723119 11572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:27.723138 11572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:27.723155 11572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:58:27.723170 11572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:58:27.728514 11572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:47707/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36473
--ranger_config_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:58:27.729507 11572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:27.730569 11572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:27.738852 11578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:27.738879 11580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:27.738852 11577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:27.739253 11572 server_base.cc:1034] running on GCE node
I20260929 00:58:27.739850 11572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:27.740929 11572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:27.742111 11572 hybrid_clock.cc:648] HybridClock initialized: now 1790643507742090 us; error 30 us; skew 500 ppm
Sep 29 00:58:27 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:27.745000 11572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:58:27.745889 11572 webserver.cc:460] Webserver started at http://127.0.98.62:34415/ using document root <none> and password file <none>
I20260929 00:58:27.746498 11572 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:27.746578 11572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:27.746806 11572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:58:27 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:58:27.802911 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "5faad67d70ed49398937ed9febd6e292"
format_stamp: "Formatted at 2026-09-29 00:58:27 on dist-test-slave-30s6"
server_key: "d40ab56c203a0004cd4c851e672daf90"
server_key_iv: "720ffb77a4c7d5caf7dde78e7804a7ef"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:27.803647 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "5faad67d70ed49398937ed9febd6e292"
format_stamp: "Formatted at 2026-09-29 00:58:27 on dist-test-slave-30s6"
server_key: "d40ab56c203a0004cd4c851e672daf90"
server_key_iv: "720ffb77a4c7d5caf7dde78e7804a7ef"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:27.809693 11572 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 00:58:27.824290 11587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:27.825711 11572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:58:27.825865 11572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "5faad67d70ed49398937ed9febd6e292"
format_stamp: "Formatted at 2026-09-29 00:58:27 on dist-test-slave-30s6"
server_key: "d40ab56c203a0004cd4c851e672daf90"
server_key_iv: "720ffb77a4c7d5caf7dde78e7804a7ef"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:27.826014 11572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:27.835309 11572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:27.838306 11572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:27.838510 11572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:27.846832 11572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36473
I20260929 00:58:27.846889 11639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36473 every 8 connection(s)
I20260929 00:58:27.847821 11572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:58:27.851272 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:27.852115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 11572
I20260929 00:58:27.852242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:58:27.852552 392 external_mini_cluster.cc:1428] Setting key fe209f460a102a2ee766af344d0785ba
I20260929 00:58:27.858361 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292: Bootstrap starting.
Sep 29 00:58:27 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:58:27.861146 11640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:27.862082 11640 log.cc:824] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:27.864416 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292: No bootstrap required, opened a new log
I20260929 00:58:27.867728 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:27.853976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.0.98.62:36473)
0929 00:58:27.854407 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:27.854432 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:27.854466 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:58:27.855098 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:27.855113 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:27.855435 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:27.855765 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:27.856545 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:27.857514 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:27.858463 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:27.858960 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:27.859134 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:27.861667 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:27.861672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:27.861685 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:27.861711 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:27.864915 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:27.865354 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:27.865367 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:27.865372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:27.865478 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:27.865786 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:27.865791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:27.865797 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:27.866165 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:27.866431 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:27.866720 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 00:58:27.867006 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":169,"threads_started":1}
I20260929 00:58:27.869282 11640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faad67d70ed49398937ed9febd6e292" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36473 } }
I20260929 00:58:27.869591 11640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:27.869654 11640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5faad67d70ed49398937ed9febd6e292, State: Initialized, Role: FOLLOWER
I20260929 00:58:27.870239 11640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faad67d70ed49398937ed9febd6e292" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36473 } }
I20260929 00:58:27.870411 11640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:58:27.870482 11640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:58:27.870584 11640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:27.871850 11640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faad67d70ed49398937ed9febd6e292" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36473 } }
I20260929 00:58:27.872236 11640 leader_election.cc:302] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5faad67d70ed49398937ed9febd6e292; no voters:
I20260929 00:58:27.872587 11640 leader_election.cc:288] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:58:27.872702 11645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:58:27.872963 11645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [term 1 LEADER]: Becoming Leader. State: Replica: 5faad67d70ed49398937ed9febd6e292, State: Running, Role: LEADER
I20260929 00:58:27.873309 11645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faad67d70ed49398937ed9febd6e292" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36473 } }
I20260929 00:58:27.873960 11640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:58:27.874899 11647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5faad67d70ed49398937ed9febd6e292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faad67d70ed49398937ed9febd6e292" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36473 } } }
I20260929 00:58:27.875030 11647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:27.875263 11647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5faad67d70ed49398937ed9febd6e292. Latest consensus state: current_term: 1 leader_uuid: "5faad67d70ed49398937ed9febd6e292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faad67d70ed49398937ed9febd6e292" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36473 } } }
I20260929 00:58:27.875339 11647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292 [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:27.876222 11654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:58:27.876536 11640 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 00:58:27.881881 11654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:58:27.889653 11654 catalog_manager.cc:1347] Generated new cluster ID: c00d52c2858a4e249fd3ec9b617b4c47
I20260929 00:58:27.889721 11654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:58:27.922495 11654 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:58:27.923862 11654 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:58:27.935983 11654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292: Generated new TSK 0
I20260929 00:58:27.936865 11654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:58:28.616762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36473
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:28.764351 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:28.764647 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:28.764724 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:28.769464 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:28.769550 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:28.769645 11685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:58:28.775724 11685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:47707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:58:28.776942 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:28.778249 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:28.794790 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:28.794893 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:28.795140 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:28.796489 11685 server_base.cc:1034] running on GCE node
I20260929 00:58:28.796967 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:28.797860 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:28.801141 11685 hybrid_clock.cc:648] HybridClock initialized: now 1790643508801053 us; error 100 us; skew 500 ppm
Sep 29 00:58:28 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:28.805699 11685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:58:28.806893 11685 webserver.cc:460] Webserver started at http://127.0.98.1:38139/ using document root <none> and password file <none>
I20260929 00:58:28.807605 11685 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:28.807698 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:28.807945 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:58:28 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:58:28.838599 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "da93200d098c4446b3009147c00634de"
format_stamp: "Formatted at 2026-09-29 00:58:28 on dist-test-slave-30s6"
server_key: "a4cfd5e9dec81e64b8dcfb638de2d370"
server_key_iv: "121fdd04d016507e80d9a6ac3d90b753"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:28.839442 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "da93200d098c4446b3009147c00634de"
format_stamp: "Formatted at 2026-09-29 00:58:28 on dist-test-slave-30s6"
server_key: "a4cfd5e9dec81e64b8dcfb638de2d370"
server_key_iv: "121fdd04d016507e80d9a6ac3d90b753"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:28.847651 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.004s
I20260929 00:58:28.886300 11702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:28.888177 11685 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.002s sys 0.001s
I20260929 00:58:28.888326 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "da93200d098c4446b3009147c00634de"
format_stamp: "Formatted at 2026-09-29 00:58:28 on dist-test-slave-30s6"
server_key: "a4cfd5e9dec81e64b8dcfb638de2d370"
server_key_iv: "121fdd04d016507e80d9a6ac3d90b753"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:28.888500 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:28.913815 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:28.917491 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:28.917686 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:28.918457 11685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:28.919529 11685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:28.919596 11685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:28.919658 11685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:28.919691 11685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:28.931967 11685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43353
I20260929 00:58:28.933107 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:58:28.937968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 11685
I20260929 00:58:28.938102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:58:28.938872 392 external_mini_cluster.cc:1428] Setting key 8ee5ffc3f4e2344e92f6d149a7c8f95a
I20260929 00:58:28.936435 11817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43353 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 00:58:28 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:58:28.961683 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:28.938447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39815 (local address 127.0.98.62:36473)
0929 00:58:28.939947 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:28.939951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:28.940407 (+ 456us) server_negotiation.cc:407] Connection header received
0929 00:58:28.941648 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:28.941654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:28.941719 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:28.941786 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:28.943588 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:28.944500 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:28.946460 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:28.946894 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:28.946988 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:28.956653 (+ 9665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:28.956659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:28.956662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:28.956696 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:28.958843 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:28.959601 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:28.959605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:28.959607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:28.959711 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:28.960185 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:28.960189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:28.960190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:28.960455 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:28.960560 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:28.961319 (+ 759us) server_negotiation.cc:299] Negotiation successful
0929 00:58:28.961469 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":56,"threads_started":1}
I20260929 00:58:28.963483 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:28.938785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36473 (local address 127.0.98.1:39815)
0929 00:58:28.940253 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:28.940308 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:28.941400 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:28.942280 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:28.942292 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:28.942671 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:28.943406 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:28.943427 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:28.944663 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:28.944669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:28.946302 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:28.946317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:28.947241 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:28.947248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:28.947478 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:28.954385 (+ 6907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:28.954417 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:28.956377 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:28.959041 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:28.959048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:28.959067 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:28.959460 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:28.959831 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:28.959834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:28.959837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:28.960040 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:28.960620 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:28.960627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:28.960992 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 00:58:28.961876 (+ 884us) client_negotiation.cc:240] Negotiation successful
0929 00:58:28.962297 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"thread_start_us":131,"threads_started":1}
I20260929 00:58:28.965628 11818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36473
I20260929 00:58:28.965893 11818 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:28.966518 11818 heartbeater.cc:511] Master 127.0.98.62:36473 requested a full tablet report, sending...
I20260929 00:58:28.968408 11604 ts_manager.cc:194] Registered new tserver with Master: da93200d098c4446b3009147c00634de (127.0.98.1:43353)
I20260929 00:58:28.970011 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:58:28.979882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36473
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 00:58:29.145320 11824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:29.146111 11824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:29.146239 11824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:29.152787 11824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:29.152871 11824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:29.152971 11824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:58:29.160832 11824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:47707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:58:29.162294 11824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:29.163718 11824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:29.176375 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:29.176369 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:29.180117 11824 server_base.cc:1034] running on GCE node
W20260929 00:58:29.182749 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:29.183284 11824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:29.184070 11824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:29.199597 11824 hybrid_clock.cc:648] HybridClock initialized: now 1790643509199546 us; error 69 us; skew 500 ppm
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:29.203678 11824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:58:29.204917 11824 webserver.cc:460] Webserver started at http://127.0.98.2:32877/ using document root <none> and password file <none>
I20260929 00:58:29.205654 11824 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:29.205756 11824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:29.206010 11824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:58:29.223778 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "9571212431f34b19b716c1b91e4067bb"
format_stamp: "Formatted at 2026-09-29 00:58:29 on dist-test-slave-30s6"
server_key: "f3d88c2c7dda9df1fd2bb9956e2c4c1d"
server_key_iv: "84c806abe5162886cd0e36e3a2284042"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:29.224546 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9571212431f34b19b716c1b91e4067bb"
format_stamp: "Formatted at 2026-09-29 00:58:29 on dist-test-slave-30s6"
server_key: "f3d88c2c7dda9df1fd2bb9956e2c4c1d"
server_key_iv: "84c806abe5162886cd0e36e3a2284042"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:29.229580 11824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:58:29.246605 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:29.248427 11824 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 00:58:29.248570 11824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "9571212431f34b19b716c1b91e4067bb"
format_stamp: "Formatted at 2026-09-29 00:58:29 on dist-test-slave-30s6"
server_key: "f3d88c2c7dda9df1fd2bb9956e2c4c1d"
server_key_iv: "84c806abe5162886cd0e36e3a2284042"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:29.248718 11824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:29.269058 11824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:29.272872 11824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:29.273075 11824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:29.273815 11824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:29.275113 11824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:29.275177 11824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:29.275231 11824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:29.275254 11824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:29.285949 11824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33589
I20260929 00:58:29.288239 11824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:58:29.290804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 11824
I20260929 00:58:29.290939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:58:29.291239 392 external_mini_cluster.cc:1428] Setting key d9f2a60657f0b7dbd70193bf44066637
I20260929 00:58:29.290079 11953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33589 every 8 connection(s)
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:58:29.318152 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.293394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32833 (local address 127.0.98.62:36473)
0929 00:58:29.293698 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:29.293703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:29.297057 (+ 3354us) server_negotiation.cc:407] Connection header received
0929 00:58:29.305031 (+ 7974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:29.305034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:29.305096 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:29.305181 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:29.306992 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.307883 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.309854 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.310279 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.310363 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.313599 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:29.313603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:29.313606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:29.313636 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:29.315413 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:29.316094 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:29.316100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:29.316105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:29.316196 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:29.316621 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:29.316628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:29.316632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:29.316904 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:29.316982 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:29.317744 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 00:58:29.317908 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260929 00:58:29.319937 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.292287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36473 (local address 127.0.98.2:32833)
0929 00:58:29.296892 (+ 4605us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:29.296955 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:29.304756 (+ 7801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:29.305549 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:29.305563 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:29.305992 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:29.306803 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.306827 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.308073 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.308077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.309680 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.309694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.310586 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.310589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.310811 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.311526 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:29.311552 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:29.313391 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:29.315546 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:29.315552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:29.315569 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:29.315974 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:29.316320 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:29.316323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:29.316325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:29.316515 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:29.317066 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:29.317076 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:29.317433 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 00:58:29.318372 (+ 939us) client_negotiation.cc:240] Negotiation successful
0929 00:58:29.318740 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4372,"thread_start_us":150,"threads_started":1}
I20260929 00:58:29.321364 11954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36473
I20260929 00:58:29.321633 11954 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:29.322288 11954 heartbeater.cc:511] Master 127.0.98.62:36473 requested a full tablet report, sending...
I20260929 00:58:29.323470 11604 ts_manager.cc:194] Registered new tserver with Master: 9571212431f34b19b716c1b91e4067bb (127.0.98.2:33589)
I20260929 00:58:29.324236 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:58:29.334985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36473
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:29.508960 11959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:29.509348 11959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:29.509505 11959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:29.516613 11959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:29.516805 11959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:29.516995 11959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:58:29.525895 11959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:47707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:58:29.527469 11959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:29.528944 11959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:29.543053 11965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:29.543565 11964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:29.544373 11967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:29.545022 11959 server_base.cc:1034] running on GCE node
I20260929 00:58:29.545573 11959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:29.546412 11959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:29.553793 11959 hybrid_clock.cc:648] HybridClock initialized: now 1790643509553754 us; error 49 us; skew 500 ppm
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:29.562824 11959 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:58:29.564090 11959 webserver.cc:460] Webserver started at http://127.0.98.3:39179/ using document root <none> and password file <none>
I20260929 00:58:29.564862 11959 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:29.564967 11959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:29.565209 11959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:58:29.584949 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbe73d20d7b44045b398bc7040b59ee0"
format_stamp: "Formatted at 2026-09-29 00:58:29 on dist-test-slave-30s6"
server_key: "d3b178b4ea6f757c5bbd0eb456612084"
server_key_iv: "0344fecbbc382cf889def85f502478e4"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:29.585764 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbe73d20d7b44045b398bc7040b59ee0"
format_stamp: "Formatted at 2026-09-29 00:58:29 on dist-test-slave-30s6"
server_key: "d3b178b4ea6f757c5bbd0eb456612084"
server_key_iv: "0344fecbbc382cf889def85f502478e4"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:29.592104 11959 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 00:58:29.617408 11974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:29.619180 11959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:58:29.619315 11959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "dbe73d20d7b44045b398bc7040b59ee0"
format_stamp: "Formatted at 2026-09-29 00:58:29 on dist-test-slave-30s6"
server_key: "d3b178b4ea6f757c5bbd0eb456612084"
server_key_iv: "0344fecbbc382cf889def85f502478e4"
server_key_version: "kuduclusterkey@0"
I20260929 00:58:29.619445 11959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:29.635905 11959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:29.639580 11959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:29.639780 11959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:29.640475 11959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:29.641484 11959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:29.641548 11959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:29.641600 11959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:29.641629 11959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:29.651826 11959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37315
I20260929 00:58:29.651937 12087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37315 every 8 connection(s)
I20260929 00:58:29.653450 11959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:58:29.654599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 11959
I20260929 00:58:29.654734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:58:29.655071 392 external_mini_cluster.cc:1428] Setting key f99b529ec0455f567197249e7c4b0aae
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:58:29.669518 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.655681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38633 (local address 127.0.98.62:36473)
0929 00:58:29.655853 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:29.655858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:29.658079 (+ 2221us) server_negotiation.cc:407] Connection header received
0929 00:58:29.659788 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:29.659791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:29.659837 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:29.659930 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:29.661266 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.661819 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.663033 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.663276 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.663388 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.665797 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:29.665801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:29.665802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:29.665825 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:29.667270 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:29.667822 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:29.667828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:29.667830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:29.667909 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:29.668307 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:29.668310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:29.668311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:29.668533 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:29.668602 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:29.669283 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 00:58:29.669405 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:58:29.670810 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.656275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36473 (local address 127.0.98.3:38633)
0929 00:58:29.657468 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:29.657529 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:29.659545 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:29.660108 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:29.660117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:29.660438 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:29.661101 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.661122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.661961 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.661964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.662905 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.662915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.663424 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.663428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.663602 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.664232 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:29.664247 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:29.665635 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:29.667401 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:29.667407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:29.667421 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:29.667728 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:29.668011 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:29.668015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:29.668016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:29.668204 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:29.668684 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:29.668692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:29.669001 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 00:58:29.669734 (+ 733us) client_negotiation.cc:240] Negotiation successful
0929 00:58:29.670013 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":174,"threads_started":1}
I20260929 00:58:29.671798 12088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36473
I20260929 00:58:29.672029 12088 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:29.672500 12088 heartbeater.cc:511] Master 127.0.98.62:36473 requested a full tablet report, sending...
I20260929 00:58:29.673465 11604 ts_manager.cc:194] Registered new tserver with Master: dbe73d20d7b44045b398bc7040b59ee0 (127.0.98.3:37315)
I20260929 00:58:29.674062 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38633
I20260929 00:58:29.680797 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:58:29.690317 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.682795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.0.98.62:36473)
0929 00:58:29.682890 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:29.682894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:29.683218 (+ 324us) server_negotiation.cc:407] Connection header received
0929 00:58:29.683353 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:29.683356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:29.683397 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:29.683481 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:29.684161 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.684671 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.685488 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.685721 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.685825 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.687219 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:29.687222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:29.687225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:29.687249 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:29.688699 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:29.689119 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:29.689124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:29.689126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:29.689214 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:29.689478 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:29.689483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:29.689485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:29.689659 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:29.689794 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:29.690023 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 00:58:29.690134 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 00:58:29.693281 11604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:58:29.696595 11604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:58:29.713584 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.705515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37315 (local address 127.0.0.1:51918)
0929 00:58:29.706925 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:29.706974 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:29.707098 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:29.707876 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:29.707883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:29.707905 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:29.708072 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.708080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.709596 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.709599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.711087 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.711094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.713057 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.713060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.713170 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.713213 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 00:58:29.713297 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 00:58:29.713416 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"thread_start_us":128,"threads_started":1}
I20260929 00:58:29.714529 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.706799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.0.98.3:37315)
0929 00:58:29.707370 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:29.707381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:29.707405 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:58:29.707479 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:29.707484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:29.707704 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:29.708230 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:29.708377 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.709478 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.711226 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.712028 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.714019 (+ 1991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.714214 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:29.714270 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 00:58:29.714376 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":89,"threads_started":1}
I20260929 00:58:29.715112 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.705913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33589 (local address 127.0.0.1:51178)
0929 00:58:29.708220 (+ 2307us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:29.708233 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:29.708303 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:29.709063 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:29.709066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:29.709081 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:29.709180 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.709186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.711457 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.711460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.712543 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.712551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.714728 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.714732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.714813 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.714834 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:58:29.714922 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:58:29.714989 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2238,"thread_start_us":74,"threads_started":1}
I20260929 00:58:29.715545 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.707127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.0.98.2:33589)
0929 00:58:29.708527 (+ 1400us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:29.708538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:29.708558 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:58:29.708625 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:29.708629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:29.708807 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:29.708937 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:29.709295 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.711342 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.712694 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.714426 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.714518 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.714673 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:29.715303 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 00:58:29.715409 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":155,"threads_started":1}
I20260929 00:58:29.718557 12021 tablet_service.cc:1467] Processing CreateTablet for tablet 453d9cf925894ae1898adea0b4caa091 (DEFAULT_TABLE table=test-table [id=c12d2cd6912c4079bbe9dd959e0d3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:29.718557 11888 tablet_service.cc:1467] Processing CreateTablet for tablet 453d9cf925894ae1898adea0b4caa091 (DEFAULT_TABLE table=test-table [id=c12d2cd6912c4079bbe9dd959e0d3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:29.719681 12021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453d9cf925894ae1898adea0b4caa091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:29.719677 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453d9cf925894ae1898adea0b4caa091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:29.729995 12107 tablet_bootstrap.cc:492] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Bootstrap starting.
I20260929 00:58:29.731911 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.706312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43353 (local address 127.0.0.1:52236)
0929 00:58:29.710912 (+ 4600us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:29.710926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:29.710999 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:29.712812 (+ 1813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:29.712815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:29.712831 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:29.712923 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.712929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.727212 (+ 14283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.727215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.728392 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.728404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.731454 (+ 3050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.731457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.731558 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.731581 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:58:29.731653 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 00:58:29.731761 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4545,"thread_start_us":76,"threads_started":1}
I20260929 00:58:29.733199 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.706467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.0.98.1:43353)
0929 00:58:29.712269 (+ 5802us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:29.712276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:29.712296 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:58:29.712361 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:29.712365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:29.712533 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:29.712677 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:29.714799 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.727061 (+ 12262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.730232 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.731120 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.731233 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.731398 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:29.732964 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0929 00:58:29.733062 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5700,"thread_start_us":78,"threads_started":1}
I20260929 00:58:29.733903 12107 tablet_bootstrap.cc:654] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:29.734781 12107 log.cc:824] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:29.736845 12107 tablet_bootstrap.cc:492] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: No bootstrap required, opened a new log
I20260929 00:58:29.736488 11752 tablet_service.cc:1467] Processing CreateTablet for tablet 453d9cf925894ae1898adea0b4caa091 (DEFAULT_TABLE table=test-table [id=c12d2cd6912c4079bbe9dd959e0d3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:29.737020 12107 ts_tablet_manager.cc:1397] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 00:58:29.737742 11752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453d9cf925894ae1898adea0b4caa091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:29.740870 12107 raft_consensus.cc:348] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.741140 12107 raft_consensus.cc:374] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:29.741250 12107 raft_consensus.cc:729] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe73d20d7b44045b398bc7040b59ee0, State: Initialized, Role: FOLLOWER
I20260929 00:58:29.741714 12107 consensus_queue.cc:260] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.744323 12107 ts_tablet_manager.cc:1428] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:58:29.744684 12088 heartbeater.cc:503] Master 127.0.98.62:36473 was elected leader, sending a full tablet report...
I20260929 00:58:29.748416 12106 tablet_bootstrap.cc:492] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: Bootstrap starting.
I20260929 00:58:29.749369 12110 tablet_bootstrap.cc:492] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de: Bootstrap starting.
I20260929 00:58:29.752072 12106 tablet_bootstrap.cc:654] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:29.752948 12106 log.cc:824] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:29.753604 12110 tablet_bootstrap.cc:654] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:29.754544 12110 log.cc:824] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:29.755208 12106 tablet_bootstrap.cc:492] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: No bootstrap required, opened a new log
I20260929 00:58:29.755412 12106 ts_tablet_manager.cc:1397] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:58:29.756778 12110 tablet_bootstrap.cc:492] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de: No bootstrap required, opened a new log
I20260929 00:58:29.756945 12110 ts_tablet_manager.cc:1397] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 00:58:29.759548 12106 raft_consensus.cc:348] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.759810 12106 raft_consensus.cc:374] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:29.759858 12106 raft_consensus.cc:729] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9571212431f34b19b716c1b91e4067bb, State: Initialized, Role: FOLLOWER
I20260929 00:58:29.760752 12106 consensus_queue.cc:260] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.761754 12106 ts_tablet_manager.cc:1428] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:58:29.761713 12110 raft_consensus.cc:348] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.762112 11954 heartbeater.cc:503] Master 127.0.98.62:36473 was elected leader, sending a full tablet report...
I20260929 00:58:29.764576 12110 raft_consensus.cc:374] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:29.764698 12110 raft_consensus.cc:729] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da93200d098c4446b3009147c00634de, State: Initialized, Role: FOLLOWER
I20260929 00:58:29.765868 12110 consensus_queue.cc:260] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.768148 11818 heartbeater.cc:503] Master 127.0.98.62:36473 was elected leader, sending a full tablet report...
I20260929 00:58:29.769492 12110 ts_tablet_manager.cc:1428] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
W20260929 00:58:29.790390 11955 tablet.cc:2369] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 00:58:29 dist-test-slave-30s6 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 00:58:29.905933 12089 tablet.cc:2369] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:58:29.940580 11819 tablet.cc:2369] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:58:29.950342 12113 raft_consensus.cc:482] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:58:29.950577 12113 raft_consensus.cc:504] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.951994 12113 leader_election.cc:288] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbe73d20d7b44045b398bc7040b59ee0 (127.0.98.3:37315), da93200d098c4446b3009147c00634de (127.0.98.1:43353)
I20260929 00:58:29.958413 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.953234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37581 (local address 127.0.98.3:37315)
0929 00:58:29.953370 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:29.953376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:29.953692 (+ 316us) server_negotiation.cc:407] Connection header received
0929 00:58:29.953916 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:29.953920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:29.953978 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:29.954055 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:29.954568 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.955353 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.957051 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.957673 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.957756 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.957825 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:29.958164 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 00:58:29.958280 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:58:29.959084 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.952407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37315 (local address 127.0.98.2:37581)
0929 00:58:29.953595 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:29.953611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:29.953777 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:29.954270 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:29.954273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:29.954294 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:29.954410 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.954418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.955563 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.955567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.956897 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.956908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.957923 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.957926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.958019 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.958043 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:58:29.958827 (+ 784us) client_negotiation.cc:240] Negotiation successful
0929 00:58:29.958920 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":107,"threads_started":1}
I20260929 00:58:29.959998 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d9cf925894ae1898adea0b4caa091" candidate_uuid: "9571212431f34b19b716c1b91e4067bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbe73d20d7b44045b398bc7040b59ee0" is_pre_election: true
I20260929 00:58:29.960356 12042 raft_consensus.cc:2454] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9571212431f34b19b716c1b91e4067bb in term 0.
I20260929 00:58:29.961333 11841 leader_election.cc:302] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9571212431f34b19b716c1b91e4067bb, dbe73d20d7b44045b398bc7040b59ee0; no voters:
I20260929 00:58:29.961686 12113 raft_consensus.cc:2781] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:58:29.961772 12113 raft_consensus.cc:482] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:58:29.961817 12113 raft_consensus.cc:3037] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:29.962596 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.953402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34437 (local address 127.0.98.1:43353)
0929 00:58:29.953504 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:29.953509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:29.955213 (+ 1704us) server_negotiation.cc:407] Connection header received
0929 00:58:29.955401 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:29.955404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:29.955455 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:29.955504 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:29.957462 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.958300 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.961216 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.961850 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.961934 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.961999 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:29.962394 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 00:58:29.962479 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 00:58:29.963002 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:29.952788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43353 (local address 127.0.98.2:34437)
0929 00:58:29.955107 (+ 2319us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:29.955123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:29.955283 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:29.957148 (+ 1865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:29.957152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:29.957172 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:29.957321 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.957326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.958921 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.958925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.961063 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:29.961075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:29.962095 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:29.962098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:29.962233 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:29.962257 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:58:29.962786 (+ 529us) client_negotiation.cc:240] Negotiation successful
0929 00:58:29.962861 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2251,"thread_start_us":90,"threads_started":1}
I20260929 00:58:29.963714 11757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d9cf925894ae1898adea0b4caa091" candidate_uuid: "9571212431f34b19b716c1b91e4067bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da93200d098c4446b3009147c00634de" is_pre_election: true
I20260929 00:58:29.963820 12113 raft_consensus.cc:504] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.964035 11757 raft_consensus.cc:2454] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9571212431f34b19b716c1b91e4067bb in term 0.
I20260929 00:58:29.964200 12113 leader_election.cc:288] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [CANDIDATE]: Term 1 election: Requested vote from peers dbe73d20d7b44045b398bc7040b59ee0 (127.0.98.3:37315), da93200d098c4446b3009147c00634de (127.0.98.1:43353)
I20260929 00:58:29.965214 11757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d9cf925894ae1898adea0b4caa091" candidate_uuid: "9571212431f34b19b716c1b91e4067bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da93200d098c4446b3009147c00634de"
I20260929 00:58:29.965343 11757 raft_consensus.cc:3037] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:29.966673 11757 raft_consensus.cc:2454] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9571212431f34b19b716c1b91e4067bb in term 1.
I20260929 00:58:29.967095 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d9cf925894ae1898adea0b4caa091" candidate_uuid: "9571212431f34b19b716c1b91e4067bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbe73d20d7b44045b398bc7040b59ee0"
I20260929 00:58:29.967253 12042 raft_consensus.cc:3037] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:29.967878 11841 leader_election.cc:302] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9571212431f34b19b716c1b91e4067bb, da93200d098c4446b3009147c00634de; no voters:
I20260929 00:58:29.968313 12113 raft_consensus.cc:2781] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:58:29.968550 12113 raft_consensus.cc:686] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 1 LEADER]: Becoming Leader. State: Replica: 9571212431f34b19b716c1b91e4067bb, State: Running, Role: LEADER
I20260929 00:58:29.968602 12042 raft_consensus.cc:2454] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9571212431f34b19b716c1b91e4067bb in term 1.
I20260929 00:58:29.968923 12113 consensus_queue.cc:237] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } }
I20260929 00:58:29.972580 11604 catalog_manager.cc:5495] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9571212431f34b19b716c1b91e4067bb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9571212431f34b19b716c1b91e4067bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9571212431f34b19b716c1b91e4067bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33589 } health_report { overall_health: HEALTHY } } }
I20260929 00:58:30.031006 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:30.025292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.0.98.2:33589)
0929 00:58:30.025433 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:30.025438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:30.025453 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:58:30.025544 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:30.025547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:30.025617 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:30.025750 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:58:30.026225 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:30.027100 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:30.028513 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:30.028888 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:30.028997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:30.029460 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:58:30.030099 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:58:30.030281 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:30.030787 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 00:58:30.030851 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:58:30.038305 11757 raft_consensus.cc:1261] T 453d9cf925894ae1898adea0b4caa091 P da93200d098c4446b3009147c00634de [term 1 FOLLOWER]: Refusing update from remote peer 9571212431f34b19b716c1b91e4067bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:58:30.038327 12042 raft_consensus.cc:1261] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 1 FOLLOWER]: Refusing update from remote peer 9571212431f34b19b716c1b91e4067bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:58:30.039179 12113 consensus_queue.cc:1035] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "da93200d098c4446b3009147c00634de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:58:30.039458 12119 consensus_queue.cc:1035] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbe73d20d7b44045b398bc7040b59ee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:58:30.048786 12122 mvcc.cc:205] Tried to move back new op lower bound from 7334475817108361216 to 7334475816836026368. Current Snapshot: MvccSnapshot[applied={T|T < 7334475817108361216}]
I20260929 00:58:30.054652 12123 mvcc.cc:205] Tried to move back new op lower bound from 7334475817108361216 to 7334475816836026368. Current Snapshot: MvccSnapshot[applied={T|T < 7334475817108361216}]
I20260929 00:58:30.067191 11604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:58:30.067386 11604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:58:30.070470 11604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5faad67d70ed49398937ed9febd6e292: Sending DeleteTablet for 3 replicas of tablet 453d9cf925894ae1898adea0b4caa091
I20260929 00:58:30.072063 12021 tablet_service.cc:1514] Processing DeleteTablet for tablet 453d9cf925894ae1898adea0b4caa091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51918
I20260929 00:58:30.072618 11752 tablet_service.cc:1514] Processing DeleteTablet for tablet 453d9cf925894ae1898adea0b4caa091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52236
I20260929 00:58:30.073102 11888 tablet_service.cc:1514] Processing DeleteTablet for tablet 453d9cf925894ae1898adea0b4caa091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51178
I20260929 00:58:30.074533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 11685
I20260929 00:58:30.082474 12131 tablet_replica.cc:330] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: stopping tablet replica
I20260929 00:58:30.082860 12131 raft_consensus.cc:2229] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:58:30.083247 12131 raft_consensus.cc:2258] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:30.087394 12131 ts_tablet_manager.cc:1905] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 00:58:30.088452 11588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43353 (error 108)
I20260929 00:58:30.089483 12133 tablet_replica.cc:330] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: stopping tablet replica
I20260929 00:58:30.089880 12133 raft_consensus.cc:2229] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:58:30.090269 12131 ts_tablet_manager.cc:1918] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:30.090368 12131 log.cc:1196] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/453d9cf925894ae1898adea0b4caa091
I20260929 00:58:30.090534 12133 raft_consensus.cc:2258] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:30.090696 12131 ts_tablet_manager.cc:1939] T 453d9cf925894ae1898adea0b4caa091 P dbe73d20d7b44045b398bc7040b59ee0: Deleting consensus metadata
I20260929 00:58:30.092144 12133 ts_tablet_manager.cc:1905] T 453d9cf925894ae1898adea0b4caa091 P 9571212431f34b19b716c1b91e4067bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:30.092284 11588 catalog_manager.cc:4841] TS dbe73d20d7b44045b398bc7040b59ee0 (127.0.98.3:37315): tablet 453d9cf925894ae1898adea0b4caa091 (table test-table [id=c12d2cd6912c4079bbe9dd959e0d3447]) successfully deleted
W20260929 00:58:30.092834 12100 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:58:30.089328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43353 (local address 127.0.0.1:52244)
0929 00:58:30.089473 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:30.089494 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:30.089593 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:30.092722 (+ 3129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 00:58:30.093094 11588 catalog_manager.cc:4584] TS da93200d098c4446b3009147c00634de (127.0.98.1:43353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 453d9cf925894ae1898adea0b4caa091: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 00:58:30.093735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 11824
I20260929 00:58:30.100157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 11959
I20260929 00:58:30.100872 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:30.100458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33589 (local address 127.0.0.1:51202)
0929 00:58:30.100691 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:30.100768 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 00:58:30.105981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 11572
I20260929 00:58:30.114437 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 00:58:30.450141 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 00:58:30.450318 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 00:58:30.795300 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 00:58:30.795 UTC [11158] LOG: received smart shutdown request
2026-09-29 00:58:30.798 UTC [11158] LOG: background worker "logical replication launcher" (PID 11167) exited with exit code 1
2026-09-29 00:58:30.799 UTC [11162] LOG: shutting down
2026-09-29 00:58:30.977 UTC [11158] LOG: database system is shut down
2026-09-29T00:58:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46852 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:58:31 dist-test-slave-30s6 krb5kdc[12146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:58:31 dist-test-slave-30s6 krb5kdc[12146](info): set up 2 sockets
Sep 29 00:58:31 dist-test-slave-30s6 krb5kdc[12146](info): commencing operation
krb5kdc: starting...
W20260929 00:58:32.146231 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:58:32 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:58:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 00:58:32.312342 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIUjItr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/postgres -l logfile start
2026-09-29 00:58:34.784 UTC [12176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 00:58:34.784 UTC [12176] LOG: listening on IPv4 address "127.0.98.20", port 59387
2026-09-29 00:58:34.787 UTC [12176] LOG: listening on Unix socket "/tmp/.s.PGSQL.59387"
2026-09-29 00:58:34.822 UTC [12179] LOG: database system was shut down at 2026-09-29 00:58:33 UTC
2026-09-29 00:58:34.826 UTC [12176] LOG: database system is ready to accept connections
I20260929 00:58:35.782091 392 mini_postgres.cc:96] Postgres bound to 59387
2026-09-29 00:58:35.788 UTC [12187] FATAL: database "slave" does not exist
127.0.98.20:59387 - accepting connections
I20260929 00:58:35.788758 392 mini_ranger.cc:161] Starting Ranger...
I20260929 00:58:35.803231 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 00:58:36.233682 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 00:58:36.233773 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-admin
2026-09-29 00:58:36,504 [I] DB FLAVOR :POSTGRES
2026-09-29 00:58:36,505 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:58:36,505 [I] Checking connection..
2026-09-29 00:58:36,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 00:58:36,759 [I] Checking connection passed.
2026-09-29 00:58:36,759 [I] --------- Verifying version history table ---------
2026-09-29 00:58:36,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:58:37,015 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:58:37,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:58:37,273 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:58:37,273 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 00:58:37,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 00:58:37.507 UTC [12251] WARNING: there is no transaction in progress
2026-09-29 00:58:37,513 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 00:58:37,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:58:37,752 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 00:58:37,752 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:58:37,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 00:58:38,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 00:58:38,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-30s6.c.gcp-upstream.internal','N') ;"
2026-09-29 00:58:38,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:58:38,771 [I] Table x_portal_user does not exist in database ranger
2026-09-29 00:58:38,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:58:39,037 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 00:58:39,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:58:39,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:58:39,513 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 00:58:39,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 00:58:40.420 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.436 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.452 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.466 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.481 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.500 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.557 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.564 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.577 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.586 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.596 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.611 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.621 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.627 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.633 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:40.640 UTC [12359] WARNING: there is no transaction in progress
2026-09-29 00:58:41,281 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 00:58:41,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 00:58:41,506 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 00:58:41,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:58:41,735 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 00:58:41,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:58:41,968 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 00:58:41,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:58:42,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:58:42,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-30s6.c.gcp-upstream.internal';"
2026-09-29 00:58:42,706 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 00:58:42,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 00:58:42,953 [I] DB_PATCHES have already been applied
I20260929 00:58:42.959417 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIUjItr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 00:58:42.959484 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 00:58:42.962903 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:45763
Sep 29, 2026 12:58:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:58:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:58:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 12:58:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 12:58:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 12:58:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 12:58:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 00:58:44.079799 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.291s user 0.006s sys 0.003s
Sep 29, 2026 12:59:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 12:59:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 12:59:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 12:59:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 12:59:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 00:59:04.329650 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 00:59:04 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643544, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 00:59:04.419526 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 00:59:04.428642 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 00:59:05.527304 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 00:59:05.527406 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms
I20260929 00:59:05.527427 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59387
2026-09-29 00:59:05,691 [I] DB FLAVOR :POSTGRES
2026-09-29 00:59:05,692 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:59:05,692 [I] Checking connection
2026-09-29 00:59:05,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 00:59:05,943 [I] connection success
2026-09-29 00:59:05,943 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 00:59:05,943 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 00:59:05,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 00:59:06,180 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 00:59:06,181 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:59:06,181 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 00:59:06,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 00:59:06,437 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 00:59:06.929373 392 mini_ranger_kms.cc:310] Created kms service
I20260929 00:59:07.095995 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 00:59:07.139424 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 00:59:07 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643544, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:07.518987 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 00:59:07.519375 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIUjItr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIUjItr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 00:59:07.519433 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 00:59:07.523262 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12539
I20260929 00:59:07.523342 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:57657
Sep 29, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 12:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 12:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 00:59:09.716389 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.297s user 0.007s sys 0.142s
I20260929 00:59:09.716521 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 00:59:09.716627 392 mini_ranger_kms.cc:401] 127.0.98.20:57657/kms/v1/keys
Sep 29 00:59:09 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643549, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 00:59:10 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643549, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 00:59:10 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643549, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 00:59:12 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:59:13.025221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36151
--ranger_config_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57657/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:13.180431 12590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:13.180807 12590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:13.180955 12590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:13.186590 12590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:13.186673 12590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:13.186702 12590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:13.186738 12590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:13.186759 12590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:13.194782 12590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:57657/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36151
--ranger_config_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:13.196089 12590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:13.197427 12590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:59:13.207346 12590 server_base.cc:1034] running on GCE node
W20260929 00:59:13.207424 12595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:13.207551 12596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:13.207424 12598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:13.208330 12590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:13.209421 12590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:13.210608 12590 hybrid_clock.cc:648] HybridClock initialized: now 1790643553210586 us; error 41 us; skew 500 ppm
Sep 29 00:59:13 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:13.213526 12590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:59:13.214439 12590 webserver.cc:460] Webserver started at http://127.0.98.62:41219/ using document root <none> and password file <none>
I20260929 00:59:13.214985 12590 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:13.215039 12590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:13.215252 12590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:59:13 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:13.265362 12590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "f0314485b7a34b518402edeecece7770"
format_stamp: "Formatted at 2026-09-29 00:59:13 on dist-test-slave-30s6"
server_key: "c4e258107bedfe6c7d45310f33ef9a28"
server_key_iv: "72f2562752425b011c7bf9794a612a05"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:13.266068 12590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0314485b7a34b518402edeecece7770"
format_stamp: "Formatted at 2026-09-29 00:59:13 on dist-test-slave-30s6"
server_key: "c4e258107bedfe6c7d45310f33ef9a28"
server_key_iv: "72f2562752425b011c7bf9794a612a05"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:13.270988 12590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:59:13.288302 12605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:13.289762 12590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:59:13.289922 12590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "f0314485b7a34b518402edeecece7770"
format_stamp: "Formatted at 2026-09-29 00:59:13 on dist-test-slave-30s6"
server_key: "c4e258107bedfe6c7d45310f33ef9a28"
server_key_iv: "72f2562752425b011c7bf9794a612a05"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:13.290062 12590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:13.303700 12590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:13.306697 12590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:13.306898 12590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:13.314879 12590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36151
I20260929 00:59:13.314891 12657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36151 every 8 connection(s)
I20260929 00:59:13.315868 12590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:13.319370 12658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:13.322976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 12590
I20260929 00:59:13.323127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:59:13.323419 392 external_mini_cluster.cc:1428] Setting key eec8723a51c7d446576f1b2519c5b002
I20260929 00:59:13.326864 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Bootstrap starting.
I20260929 00:59:13.329788 12658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:13.330834 12658 log.cc:824] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:59:13 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:13.332903 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: No bootstrap required, opened a new log
I20260929 00:59:13.337423 12658 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } }
I20260929 00:59:13.337693 12658 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:13.337787 12658 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0314485b7a34b518402edeecece7770, State: Initialized, Role: FOLLOWER
I20260929 00:59:13.338284 12658 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } }
I20260929 00:59:13.338434 12658 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:13.338496 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:13.324849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.0.98.62:36151)
0929 00:59:13.325241 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:13.325258 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:13.325291 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:59:13.325914 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:13.325931 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:13.326333 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:13.326671 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:13.327466 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:13.328420 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:13.329298 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:13.329763 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:13.329942 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:13.332902 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:13.332908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:13.332925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:13.332957 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:13.335024 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:13.335574 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:13.335580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:13.335586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:13.335732 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:13.336059 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:13.336062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:13.336064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:13.336567 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:13.336768 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:13.337086 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 00:59:13.337535 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260929 00:59:13.338526 12658 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:13.338871 12658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:13.339991 12658 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } }
I20260929 00:59:13.340348 12658 leader_election.cc:302] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0314485b7a34b518402edeecece7770; no voters:
I20260929 00:59:13.340746 12658 leader_election.cc:288] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:13.340842 12663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:13.341091 12663 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 1 LEADER]: Becoming Leader. State: Replica: f0314485b7a34b518402edeecece7770, State: Running, Role: LEADER
I20260929 00:59:13.341427 12663 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } }
I20260929 00:59:13.342027 12658 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:13.342623 12665 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0314485b7a34b518402edeecece7770. Latest consensus state: current_term: 1 leader_uuid: "f0314485b7a34b518402edeecece7770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } } }
I20260929 00:59:13.342978 12665 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:13.342849 12664 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0314485b7a34b518402edeecece7770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } } }
I20260929 00:59:13.343573 12664 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:13.344585 12672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:13.344772 12658 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 00:59:13.350935 12672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:13.362609 12672 catalog_manager.cc:1347] Generated new cluster ID: d652c0525d9e446698fd3d72e8c2fe9f
I20260929 00:59:13.362690 12672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:13.394631 12672 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:13.396070 12672 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:13.414083 12672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Generated new TSK 0
I20260929 00:59:13.415057 12672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:59:14.159022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36151
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:14.296973 12703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:14.297262 12703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:14.297338 12703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:14.301915 12703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:14.301999 12703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:14.302109 12703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:59:14.308122 12703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:57657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:14.309226 12703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:14.310405 12703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:14.321452 12711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:14.321451 12713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:14.321718 12710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:14.322942 12703 server_base.cc:1034] running on GCE node
I20260929 00:59:14.323422 12703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:14.324237 12703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:14.325407 12703 hybrid_clock.cc:648] HybridClock initialized: now 1790643554325390 us; error 32 us; skew 500 ppm
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:14.328859 12703 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:59:14.329977 12703 webserver.cc:460] Webserver started at http://127.0.98.1:40393/ using document root <none> and password file <none>
I20260929 00:59:14.330694 12703 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:14.330793 12703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:14.331020 12703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:14.347262 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "f1503504d164461aa2085289744eafbd"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-30s6"
server_key: "58f0216ee09f3ec0f3865feea42eaca2"
server_key_iv: "bc6fec21b6f4276855a2f29434121fc2"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:14.348016 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f1503504d164461aa2085289744eafbd"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-30s6"
server_key: "58f0216ee09f3ec0f3865feea42eaca2"
server_key_iv: "bc6fec21b6f4276855a2f29434121fc2"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:14.357263 12703 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260929 00:59:14.382566 12720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:14.384454 12703 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260929 00:59:14.384588 12703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "f1503504d164461aa2085289744eafbd"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-30s6"
server_key: "58f0216ee09f3ec0f3865feea42eaca2"
server_key_iv: "bc6fec21b6f4276855a2f29434121fc2"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:14.384749 12703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:14.405843 12703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:14.409595 12703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:14.409857 12703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:14.410663 12703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:14.411775 12703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:14.411880 12703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:14.412004 12703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:14.412088 12703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:14.423735 12703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46681
I20260929 00:59:14.424639 12703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:14.425177 12835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46681 every 8 connection(s)
I20260929 00:59:14.426967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 12703
I20260929 00:59:14.427091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:59:14.427402 392 external_mini_cluster.cc:1428] Setting key 72da0b44cab514ead9ac75c48e048688
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 00:59:14.442768 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:14.427869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59543 (local address 127.0.98.62:36151)
0929 00:59:14.430228 (+ 2359us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:14.430235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:14.430252 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:59:14.430294 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:14.430300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:14.430358 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:14.430445 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:14.431732 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.432627 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.434506 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.434915 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.436890 (+ 1975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:14.438067 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:14.438074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:14.438078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:14.438111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:14.439962 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:14.440520 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:14.440525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:14.440527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:14.440612 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:14.440968 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:14.440971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:14.440973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:14.441197 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:14.441293 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:14.442352 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0929 00:59:14.442488 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2243,"thread_start_us":81,"threads_started":1}
I20260929 00:59:14.444783 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:14.427368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36151 (local address 127.0.98.1:59543)
0929 00:59:14.428591 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:14.428647 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:14.429794 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:14.430793 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:14.430800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:14.431054 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:14.431580 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:14.431595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.432789 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.432796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:14.434333 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:14.434348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.435092 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.435098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:14.435339 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:14.436093 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:14.436120 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:14.437869 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:14.440088 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:14.440093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:14.440108 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:14.440416 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:14.440722 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:14.440725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:14.440727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:14.440864 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:14.441291 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:14.441298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:14.441550 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 00:59:14.441753 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 00:59:14.443737 (+ 1984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"thread_start_us":146,"threads_started":1}
I20260929 00:59:14.446084 12836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36151
I20260929 00:59:14.446349 12836 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:14.446940 12836 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
I20260929 00:59:14.448802 12622 ts_manager.cc:194] Registered new tserver with Master: f1503504d164461aa2085289744eafbd (127.0.98.1:46681)
I20260929 00:59:14.450425 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:59:14.459347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36151
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 00:59:14.585515 12842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:14.585803 12842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:14.585937 12842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:14.590036 12842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:14.590122 12842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:14.590235 12842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:59:14.595918 12842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:57657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:14.596977 12842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:14.598248 12842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:14.607965 12848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:14.608207 12849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:14.608057 12851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:14.609586 12842 server_base.cc:1034] running on GCE node
I20260929 00:59:14.610023 12842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:14.610770 12842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:14.611984 12842 hybrid_clock.cc:648] HybridClock initialized: now 1790643554611955 us; error 50 us; skew 500 ppm
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:14.624191 12842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:59:14.625334 12842 webserver.cc:460] Webserver started at http://127.0.98.2:39657/ using document root <none> and password file <none>
I20260929 00:59:14.625922 12842 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:14.626014 12842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:14.626289 12842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:14.639405 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "fda39cc835844a4599a57d6399d22c39"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-30s6"
server_key: "d436b8bd2b2d07255edf7ba9a2132362"
server_key_iv: "2c507842649af5aa5586ef71f8446d55"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:14.640208 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fda39cc835844a4599a57d6399d22c39"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-30s6"
server_key: "d436b8bd2b2d07255edf7ba9a2132362"
server_key_iv: "2c507842649af5aa5586ef71f8446d55"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:14.645145 12842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:59:14.670122 12858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:14.671861 12842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 00:59:14.671991 12842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "fda39cc835844a4599a57d6399d22c39"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-30s6"
server_key: "d436b8bd2b2d07255edf7ba9a2132362"
server_key_iv: "2c507842649af5aa5586ef71f8446d55"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:14.672120 12842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:14.697140 12842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:14.700937 12842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:14.701200 12842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:14.701924 12842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:14.703018 12842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:14.703125 12842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:14.703238 12842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:14.703305 12842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:14.713753 12842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38451
I20260929 00:59:14.719460 12842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:14.723579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 12842
I20260929 00:59:14.723722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:59:14.724047 392 external_mini_cluster.cc:1428] Setting key fe1c929701072d0f74f5518388390948
I20260929 00:59:14.721217 12971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38451 every 8 connection(s)
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:14.740100 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:14.724759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54397 (local address 127.0.98.62:36151)
0929 00:59:14.724934 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:14.724939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:14.726327 (+ 1388us) server_negotiation.cc:407] Connection header received
0929 00:59:14.727822 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:14.727827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:14.727877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:14.727928 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:14.729553 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.730407 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.732049 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.732385 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.732456 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:14.735526 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:14.735531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:14.735536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:14.735570 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:14.737473 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:14.738123 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:14.738127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:14.738129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:14.738249 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:14.738728 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:14.738745 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:14.738747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:14.738969 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:14.739044 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:14.739789 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 00:59:14.739929 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:59:14.744211 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:14.723567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36151 (local address 127.0.98.2:54397)
0929 00:59:14.726121 (+ 2554us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:14.726229 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:14.727564 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:14.728344 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:14.728354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:14.728705 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:14.729378 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:14.729398 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.730562 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.730565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:14.731916 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:14.731926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.732648 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.732651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:14.732833 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:14.733538 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:14.733559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:14.735322 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:14.737625 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:14.737632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:14.737646 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:14.737995 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:14.738373 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:14.738376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:14.738378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:14.738556 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:14.739131 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:14.739137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:14.739471 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 00:59:14.742110 (+ 2639us) client_negotiation.cc:240] Negotiation successful
0929 00:59:14.743027 (+ 917us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2314,"thread_start_us":164,"threads_started":1}
I20260929 00:59:14.746047 12972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36151
I20260929 00:59:14.746309 12972 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:14.746891 12972 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
I20260929 00:59:14.748018 12622 ts_manager.cc:194] Registered new tserver with Master: fda39cc835844a4599a57d6399d22c39 (127.0.98.2:38451)
I20260929 00:59:14.748759 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:59:14.769429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36151
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:14.939320 12977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:14.939695 12977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:14.939769 12977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:14.946002 12977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:14.946112 12977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:14.946250 12977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:59:14.954403 12977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:57657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:14.956028 12977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:14.957495 12977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:14.973253 12983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:14.973814 12985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:14.974216 12982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:14.975669 12977 server_base.cc:1034] running on GCE node
I20260929 00:59:14.976135 12977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:14.976953 12977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:14.985817 12977 hybrid_clock.cc:648] HybridClock initialized: now 1790643554985727 us; error 94 us; skew 500 ppm
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:14.994012 12977 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:59:14.995333 12977 webserver.cc:460] Webserver started at http://127.0.98.3:33653/ using document root <none> and password file <none>
I20260929 00:59:14.996189 12977 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:14.996342 12977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:14.996639 12977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:59:14 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:15.013213 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "5d5538978c764c51ba60d723e9d919cd"
format_stamp: "Formatted at 2026-09-29 00:59:15 on dist-test-slave-30s6"
server_key: "0646c783e0578e7cae31095035e7c8ef"
server_key_iv: "05e42bc1a9dd71fe6feaf40d4c5486ee"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:15.013839 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5d5538978c764c51ba60d723e9d919cd"
format_stamp: "Formatted at 2026-09-29 00:59:15 on dist-test-slave-30s6"
server_key: "0646c783e0578e7cae31095035e7c8ef"
server_key_iv: "05e42bc1a9dd71fe6feaf40d4c5486ee"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:15.026886 12977 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.000s
I20260929 00:59:15.059378 12992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.063426 12977 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260929 00:59:15.063576 12977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "5d5538978c764c51ba60d723e9d919cd"
format_stamp: "Formatted at 2026-09-29 00:59:15 on dist-test-slave-30s6"
server_key: "0646c783e0578e7cae31095035e7c8ef"
server_key_iv: "05e42bc1a9dd71fe6feaf40d4c5486ee"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:15.063733 12977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:15.090049 12977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:15.093995 12977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:15.094306 12977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:15.095095 12977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:15.096273 12977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:15.096385 12977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.096483 12977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:15.096553 12977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.111712 12977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34335
I20260929 00:59:15.112948 12977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:15.117125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 12977
I20260929 00:59:15.117269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:59:15.117588 392 external_mini_cluster.cc:1428] Setting key 2c6ceda9ca7da456841b237a1fcde2c5
I20260929 00:59:15.114295 13105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34335 every 8 connection(s)
Sep 29 00:59:15 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:15.138149 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:15.117499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38785 (local address 127.0.98.62:36151)
0929 00:59:15.117649 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:15.117654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:15.118686 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 00:59:15.119941 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:15.119945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:15.119996 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:15.120046 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:15.121700 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:15.122545 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:15.124400 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:15.124746 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:15.124820 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:15.127879 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:15.127883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:15.127886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:15.127914 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:15.129757 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:15.130414 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:15.130418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:15.130419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:15.130507 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:15.130929 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:15.130933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:15.130934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:15.131169 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:15.131250 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:15.137815 (+ 6565us) server_negotiation.cc:299] Negotiation successful
0929 00:59:15.137972 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:59:15.139091 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:15.116411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36151 (local address 127.0.98.3:38785)
0929 00:59:15.118533 (+ 2122us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:15.118596 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:15.119705 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:15.120476 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:15.120486 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:15.120846 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:15.121527 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:15.121548 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:15.122825 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:15.122828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:15.124261 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:15.124272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:15.125017 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:15.125020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:15.125222 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:15.125936 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:15.125956 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:15.127695 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:15.129894 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:15.129900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:15.129916 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:15.130309 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:15.130623 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:15.130626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:15.130628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:15.130813 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:15.131320 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:15.131327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:15.137398 (+ 6071us) client_negotiation.cc:769] Sending connection context
0929 00:59:15.137724 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 00:59:15.138093 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":157,"threads_started":1}
I20260929 00:59:15.140383 13106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36151
I20260929 00:59:15.140654 13106 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:15.141178 13106 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
I20260929 00:59:15.142400 12622 ts_manager.cc:194] Registered new tserver with Master: 5d5538978c764c51ba60d723e9d919cd (127.0.98.3:34335)
I20260929 00:59:15.143154 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38785
I20260929 00:59:15.143375 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:15.144083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 12703
I20260929 00:59:15.150390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 12842
I20260929 00:59:15.155905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 12977
I20260929 00:59:15.162163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 12590
I20260929 00:59:15.168519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36151
--webserver_interface=127.0.98.62
--webserver_port=41219
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36151
--ranger_config_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57657/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57657/kms with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:15.279366 13109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:15.279645 13109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:15.279692 13109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:15.283617 13109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:15.283671 13109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:15.283696 13109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:15.283712 13109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:15.283728 13109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:15.289234 13109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57657/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36151
--ranger_config_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:15.290282 13109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:15.291378 13109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:15.300074 13114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:15.300313 13115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:15.300326 13117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:15.300467 13109 server_base.cc:1034] running on GCE node
I20260929 00:59:15.301090 13109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:15.302307 13109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:15.303485 13109 hybrid_clock.cc:648] HybridClock initialized: now 1790643555303470 us; error 30 us; skew 500 ppm
Sep 29 00:59:15 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:15.306159 13109 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:59:15.307117 13109 webserver.cc:460] Webserver started at http://127.0.98.62:41219/ using document root <none> and password file <none>
I20260929 00:59:15.307646 13109 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:15.307751 13109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:59:15 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:15.379762 13109 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:59:15.382552 13125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.384192 13109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:15.384332 13109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "f0314485b7a34b518402edeecece7770"
format_stamp: "Formatted at 2026-09-29 00:59:13 on dist-test-slave-30s6"
server_key: "c4e258107bedfe6c7d45310f33ef9a28"
server_key_iv: "72f2562752425b011c7bf9794a612a05"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:15.385061 13109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:15.410328 13109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:15.413462 13109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:15.413625 13109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:15.420914 13177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36151 every 8 connection(s)
I20260929 00:59:15.420912 13109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36151
I20260929 00:59:15.422138 13109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:15.426321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 13109
I20260929 00:59:15.426471 13178 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:59:15.426785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46681
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36151
--webserver_port=40393
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57657/kms with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:15.428221 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Bootstrap starting.
I20260929 00:59:15.444756 13178 log.cc:824] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:15.451359 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:59:15.451771 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Bootstrap complete.
I20260929 00:59:15.455862 13178 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } }
I20260929 00:59:15.456262 13178 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0314485b7a34b518402edeecece7770, State: Initialized, Role: FOLLOWER
I20260929 00:59:15.456744 13178 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } }
I20260929 00:59:15.456883 13178 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:15.456970 13178 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:15.457088 13178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:59:15.459079 13178 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } }
I20260929 00:59:15.459405 13178 leader_election.cc:302] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0314485b7a34b518402edeecece7770; no voters:
I20260929 00:59:15.459720 13178 leader_election.cc:288] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:59:15.459863 13182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:59:15.460165 13182 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [term 2 LEADER]: Becoming Leader. State: Replica: f0314485b7a34b518402edeecece7770, State: Running, Role: LEADER
I20260929 00:59:15.460527 13182 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } }
I20260929 00:59:15.461014 13178 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:15.461544 13182 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0314485b7a34b518402edeecece7770. Latest consensus state: current_term: 2 leader_uuid: "f0314485b7a34b518402edeecece7770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } } }
I20260929 00:59:15.461740 13182 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:15.461942 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f0314485b7a34b518402edeecece7770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0314485b7a34b518402edeecece7770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36151 } } }
I20260929 00:59:15.462054 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:15.463389 13187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:15.463439 13178 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 00:59:15.469343 13187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:15.470248 13187 catalog_manager.cc:1259] Loaded cluster ID: d652c0525d9e446698fd3d72e8c2fe9f
I20260929 00:59:15.470362 13187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:15.474014 13187 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:15.474831 13187 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Loaded TSK: 0
I20260929 00:59:15.476133 13187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:59:15.573764 13180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:15.574208 13180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:15.574328 13180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:15.580745 13180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:15.580909 13180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:15.581050 13180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:59:15.589346 13180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40393
--tserver_master_addrs=127.0.98.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:15.590905 13180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:15.592242 13180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:15.604869 13218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:15.605475 13180 server_base.cc:1034] running on GCE node
W20260929 00:59:15.605688 13215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:15.606256 13216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:15.606746 13180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:15.607540 13180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:15.608737 13180 hybrid_clock.cc:648] HybridClock initialized: now 1790643555608709 us; error 41 us; skew 500 ppm
Sep 29 00:59:15 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:15.612313 13180 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:59:15.613507 13180 webserver.cc:460] Webserver started at http://127.0.98.1:40393/ using document root <none> and password file <none>
I20260929 00:59:15.614224 13180 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:15.614351 13180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:59:15 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:15.634771 13180 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:59:15.637946 13226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.639686 13180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:15.639813 13180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "f1503504d164461aa2085289744eafbd"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-30s6"
server_key: "58f0216ee09f3ec0f3865feea42eaca2"
server_key_iv: "bc6fec21b6f4276855a2f29434121fc2"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:15.640540 13180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:15.671742 13180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:15.675539 13180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:15.675814 13180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:15.676551 13180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:15.677600 13180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:15.677666 13180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.677722 13180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:15.677796 13180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.689183 13180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46681
I20260929 00:59:15.689759 13339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46681 every 8 connection(s)
I20260929 00:59:15.690711 13180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:15.699250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 13180
I20260929 00:59:15.699803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38451
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36151
--webserver_port=39657
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57657/kms with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:59:15 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:15.714154 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:15.692970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40677 (local address 127.0.98.62:36151)
0929 00:59:15.693564 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:15.693588 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:15.694273 (+ 685us) server_negotiation.cc:407] Connection header received
0929 00:59:15.695325 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:15.695335 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:15.695637 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:15.695967 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:15.697133 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:15.698594 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:15.700374 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:15.701123 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:15.701317 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:15.708531 (+ 7214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:15.708537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:15.708554 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:15.708585 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:15.710676 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:15.711238 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:15.711245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:15.711250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:15.711372 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:15.711773 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:15.711778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:15.711779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:15.712173 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:15.712315 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:15.712891 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 00:59:15.713134 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":117,"threads_started":1}
I20260929 00:59:15.714197 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:15.693141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36151 (local address 127.0.98.1:40677)
0929 00:59:15.693855 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:15.693908 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:15.694865 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:15.696045 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:15.696053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:15.696391 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:15.696955 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:15.696971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:15.698777 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:15.698781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:15.700202 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:15.700214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:15.701538 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:15.701542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:15.701741 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:15.706395 (+ 4654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:15.706421 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:15.708273 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:15.710811 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:15.710817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:15.710832 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:15.711101 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:15.711471 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:15.711475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:15.711477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:15.711677 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:15.712302 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:15.712309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:15.712576 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 00:59:15.712767 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 00:59:15.713109 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":115,"threads_started":1}
I20260929 00:59:15.717150 13340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36151
I20260929 00:59:15.717398 13340 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:15.717914 13340 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
W20260929 00:59:15.720108 13340 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36151 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 00:59:15.720248 13340 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:15.720415 13340 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
I20260929 00:59:15.721220 13340 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:15.721411 13340 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
W20260929 00:59:15.722142 13340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:59:15.879125 13344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:15.879410 13344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:15.879500 13344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:15.883863 13344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:15.883940 13344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:15.884027 13344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:59:15.889528 13344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39657
--tserver_master_addrs=127.0.98.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:15.891031 13344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:15.892329 13344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:15.903963 13352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:15.903963 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:15.905691 13350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:15.906165 13344 server_base.cc:1034] running on GCE node
I20260929 00:59:15.906757 13344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:15.907668 13344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:15.908938 13344 hybrid_clock.cc:648] HybridClock initialized: now 1790643555908919 us; error 35 us; skew 500 ppm
Sep 29 00:59:15 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:15.912662 13344 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:59:15.913802 13344 webserver.cc:460] Webserver started at http://127.0.98.2:39657/ using document root <none> and password file <none>
I20260929 00:59:15.914597 13344 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:15.914757 13344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:59:15 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:15.943121 13344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:59:15.945880 13360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.947481 13344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:15.947628 13344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "fda39cc835844a4599a57d6399d22c39"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-30s6"
server_key: "d436b8bd2b2d07255edf7ba9a2132362"
server_key_iv: "2c507842649af5aa5586ef71f8446d55"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:15.948205 13344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:15.981376 13344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:15.985239 13344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:15.985463 13344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:15.986238 13344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:15.987300 13344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:15.987365 13344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.987437 13344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:15.987471 13344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:15.998586 13344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38451
I20260929 00:59:15.998606 13474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38451 every 8 connection(s)
I20260929 00:59:15.999938 13344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:16.006760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 13344
I20260929 00:59:16.007304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34335
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36151
--webserver_port=33653
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57657/kms with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:59:16 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:16.019579 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:16.003716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52309 (local address 127.0.98.62:36151)
0929 00:59:16.003882 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:16.003888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:16.004372 (+ 484us) server_negotiation.cc:407] Connection header received
0929 00:59:16.005622 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:16.005626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:16.005684 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:16.005790 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:16.007341 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.008173 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.009865 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.010229 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.010308 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:16.014487 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:16.014491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:16.014495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:16.014526 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:16.016700 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:16.017364 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:16.017368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:16.017370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:16.017463 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:16.017899 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:16.017902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:16.017904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:16.018153 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:16.018295 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:16.019228 (+ 933us) server_negotiation.cc:299] Negotiation successful
0929 00:59:16.019381 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:59:16.021341 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:16.002628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36151 (local address 127.0.98.2:52309)
0929 00:59:16.004165 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:16.004231 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:16.005358 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:16.005969 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:16.005980 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:16.006425 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:16.007160 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:16.007180 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.008344 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.008347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:16.009727 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:16.009737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.010515 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.010519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:16.010731 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:16.011423 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:16.011446 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:16.013369 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:16.016851 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:16.016857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:16.016877 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:16.017243 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:16.017583 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:16.017587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:16.017589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:16.017777 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:16.018364 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:16.018373 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:16.018909 (+ 536us) client_negotiation.cc:769] Sending connection context
0929 00:59:16.019855 (+ 946us) client_negotiation.cc:240] Negotiation successful
0929 00:59:16.020213 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"thread_start_us":124,"threads_started":1}
I20260929 00:59:16.022828 13475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36151
I20260929 00:59:16.023087 13475 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:16.023643 13475 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
W20260929 00:59:16.025187 13475 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36151 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 00:59:16.025296 13475 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:16.025465 13475 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
I20260929 00:59:16.026263 13475 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:16.026422 13475 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
W20260929 00:59:16.027122 13475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:59:16.149154 13478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:16.149498 13478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:16.149598 13478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:16.154016 13478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:16.154109 13478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:16.154246 13478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:59:16.159886 13478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33653
--tserver_master_addrs=127.0.98.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:16.161098 13478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:16.162413 13478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:16.177943 13484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:16.177968 13483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:16.177943 13486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:16.178558 13478 server_base.cc:1034] running on GCE node
I20260929 00:59:16.179121 13478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:16.180004 13478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:16.181200 13478 hybrid_clock.cc:648] HybridClock initialized: now 1790643556181162 us; error 59 us; skew 500 ppm
Sep 29 00:59:16 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:16.184514 13478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:59:16.186391 13478 webserver.cc:460] Webserver started at http://127.0.98.3:33653/ using document root <none> and password file <none>
I20260929 00:59:16.187167 13478 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:16.187345 13478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:59:16 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:16.208770 13478 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:59:16.212204 13494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:16.213939 13478 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 00:59:16.214080 13478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "5d5538978c764c51ba60d723e9d919cd"
format_stamp: "Formatted at 2026-09-29 00:59:15 on dist-test-slave-30s6"
server_key: "0646c783e0578e7cae31095035e7c8ef"
server_key_iv: "05e42bc1a9dd71fe6feaf40d4c5486ee"
server_key_version: "kuduclusterkey@0"
I20260929 00:59:16.214825 13478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:16.233860 13478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:16.237110 13478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:16.237320 13478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:16.237936 13478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:16.238855 13478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:16.238926 13478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:16.238996 13478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:16.239033 13478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:16.249797 13478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34335
I20260929 00:59:16.249962 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34335 every 8 connection(s)
I20260929 00:59:16.251024 13478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:16.256336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 13478
Sep 29 00:59:16 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:16.268594 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:16.253262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45765 (local address 127.0.98.62:36151)
0929 00:59:16.254366 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:16.254371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:16.254560 (+ 189us) server_negotiation.cc:407] Connection header received
0929 00:59:16.255767 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:16.255771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:16.255826 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:16.255923 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:16.257528 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.258412 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.260063 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.260406 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.260514 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:16.264376 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:16.264380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:16.264382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:16.264410 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:16.266266 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:16.266866 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:16.266870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:16.266872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:16.266969 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:16.267387 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:16.267390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:16.267392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:16.267630 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:16.267741 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:16.268266 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 00:59:16.268396 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020}
I20260929 00:59:16.269353 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:16.253808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36151 (local address 127.0.98.3:45765)
0929 00:59:16.254388 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:16.254449 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:16.255528 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:16.256257 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:16.256266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:16.256626 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:16.257339 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:16.257360 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.258567 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.258570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:16.259899 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:16.259910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.260682 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.260686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:16.260888 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:16.261739 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:16.261764 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:16.264183 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:16.266392 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:16.266398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:16.266416 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:16.266750 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:16.267091 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:16.267097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:16.267099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:16.267288 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:16.267738 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:16.267743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:16.267994 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 00:59:16.268179 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 00:59:16.268428 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":140,"threads_started":1}
I20260929 00:59:16.270825 13608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36151
I20260929 00:59:16.271090 13608 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:16.271678 13608 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
I20260929 00:59:16.273180 13142 ts_manager.cc:194] Registered new tserver with Master: 5d5538978c764c51ba60d723e9d919cd (127.0.98.3:34335)
I20260929 00:59:16.273311 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:16.257343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.0.98.62:36151)
0929 00:59:16.258812 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:16.258816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:16.258829 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:59:16.258865 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:16.258868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:16.258913 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:16.259005 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:16.260530 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.261270 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.263824 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.264128 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.264246 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:16.265790 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:16.265794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:16.265797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:16.265844 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:16.271429 (+ 5585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:16.271906 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:16.271913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:16.271915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:16.272006 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:16.272329 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:16.272333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:16.272336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:16.272565 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:16.272757 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:16.272946 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 00:59:16.273070 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":91,"threads_started":1}
I20260929 00:59:16.274662 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45765
Sep 29 00:59:16 dist-test-slave-30s6 krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:16.722407 13340 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:16.722738 13340 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
I20260929 00:59:16.723515 13142 ts_manager.cc:194] Registered new tserver with Master: f1503504d164461aa2085289744eafbd (127.0.98.1:46681)
I20260929 00:59:16.724316 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40677
Sep 29 00:59:16 dist-test-slave-30s6 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 00:59:17.027361 13475 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:17.027665 13475 heartbeater.cc:511] Master 127.0.98.62:36151 requested a full tablet report, sending...
I20260929 00:59:17.028498 13142 ts_manager.cc:194] Registered new tserver with Master: fda39cc835844a4599a57d6399d22c39 (127.0.98.2:38451)
I20260929 00:59:17.029248 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52309
I20260929 00:59:17.033272 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:17.045675 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.035141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.0.98.62:36151)
0929 00:59:17.035866 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.035872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:17.035889 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:59:17.035937 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:17.035942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:17.035999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:17.036937 (+ 938us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:17.037035 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.037854 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.039448 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.039798 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.040262 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.041514 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:17.041517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:17.041520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:17.041545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:17.043463 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:17.043974 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:17.043977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:17.043979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:17.044077 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:17.044368 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:17.044371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:17.044373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:17.044580 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:17.045288 (+ 708us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:17.045366 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 00:59:17.045492 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":87,"threads_started":1}
I20260929 00:59:17.050099 13142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:59:17.055513 13142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:59:17.075368 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.066170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38451 (local address 127.0.0.1:42940)
0929 00:59:17.070756 (+ 4586us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:17.070765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:17.070866 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:17.071293 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:17.071299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:17.071321 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:17.071469 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.071476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.072449 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.072455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.073644 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.073653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.074761 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.074764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.074899 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.074967 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 00:59:17.075073 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 00:59:17.075196 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4050,"mutex_wait_us":16,"thread_start_us":91,"threads_started":1}
I20260929 00:59:17.075986 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.066416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.0.98.2:38451)
0929 00:59:17.070372 (+ 3956us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.070379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:17.070858 (+ 479us) server_negotiation.cc:407] Connection header received
0929 00:59:17.070971 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:17.070977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:17.071155 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:17.071307 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:17.071594 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.072337 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.073763 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.074432 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.074551 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.074696 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:17.075748 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0929 00:59:17.075850 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3848,"thread_start_us":98,"threads_started":1}
I20260929 00:59:17.079279 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.067347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46681 (local address 127.0.0.1:37540)
0929 00:59:17.074316 (+ 6969us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:17.074330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:17.074500 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:17.075150 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:17.075153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:17.075169 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:17.075248 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.075253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.076592 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.076595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.077651 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.077661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.078891 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.078895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.078979 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.079003 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:59:17.079073 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 00:59:17.079139 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6908,"thread_start_us":90,"threads_started":1}
I20260929 00:59:17.079681 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.067488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.0.98.1:46681)
0929 00:59:17.067816 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.067823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:17.074427 (+ 6604us) server_negotiation.cc:407] Connection header received
0929 00:59:17.074656 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:17.074662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:17.074863 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:17.074971 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:17.075367 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.076446 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.077788 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.078592 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.078697 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.078829 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:17.079443 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 00:59:17.079553 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":123,"threads_started":1}
I20260929 00:59:17.079241 13409 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8fe18d216048caa3e1b8dc731b6721 (DEFAULT_TABLE table=test-table [id=ed5e19f48a76468bb2a8f65a68e8f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:17.080955 13409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8fe18d216048caa3e1b8dc731b6721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:17.082870 13273 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8fe18d216048caa3e1b8dc731b6721 (DEFAULT_TABLE table=test-table [id=ed5e19f48a76468bb2a8f65a68e8f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:17.083932 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8fe18d216048caa3e1b8dc731b6721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:17.093417 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.066075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34335 (local address 127.0.0.1:55206)
0929 00:59:17.070455 (+ 4380us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:17.070491 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:17.070596 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:17.074884 (+ 4288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:17.074887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:17.074903 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:17.075025 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.075035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.086525 (+ 11490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.086529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.087707 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.087720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.092934 (+ 5214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.092938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.093069 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.093111 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 00:59:17.093166 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 00:59:17.093261 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4218,"mutex_wait_us":5,"thread_start_us":124,"threads_started":1}
I20260929 00:59:17.094352 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.066909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.0.98.3:34335)
0929 00:59:17.074416 (+ 7507us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.074423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:17.074449 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:59:17.074528 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:17.074533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:17.074742 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:17.075175 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:17.075305 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.076769 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.088962 (+ 12193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.089817 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.093798 (+ 3981us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.093985 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:17.094058 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 00:59:17.094157 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7391,"thread_start_us":87,"threads_started":1}
I20260929 00:59:17.097746 13542 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8fe18d216048caa3e1b8dc731b6721 (DEFAULT_TABLE table=test-table [id=ed5e19f48a76468bb2a8f65a68e8f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:17.098961 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8fe18d216048caa3e1b8dc731b6721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:17.102622 13642 tablet_bootstrap.cc:492] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: Bootstrap starting.
I20260929 00:59:17.105878 13642 tablet_bootstrap.cc:654] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:17.106769 13642 log.cc:824] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:17.110570 13641 tablet_bootstrap.cc:492] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd: Bootstrap starting.
I20260929 00:59:17.112454 13641 tablet_bootstrap.cc:654] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:17.113175 13641 log.cc:824] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:17.113674 13644 tablet_bootstrap.cc:492] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: Bootstrap starting.
I20260929 00:59:17.115481 13641 tablet_bootstrap.cc:492] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd: No bootstrap required, opened a new log
I20260929 00:59:17.115778 13641 ts_tablet_manager.cc:1397] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:59:17.116516 13642 tablet_bootstrap.cc:492] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: No bootstrap required, opened a new log
I20260929 00:59:17.116840 13642 ts_tablet_manager.cc:1397] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260929 00:59:17.117486 13644 tablet_bootstrap.cc:654] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:17.118368 13644 log.cc:824] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:17.120221 13641 raft_consensus.cc:348] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.120613 13641 raft_consensus.cc:374] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:17.120720 13641 raft_consensus.cc:729] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1503504d164461aa2085289744eafbd, State: Initialized, Role: FOLLOWER
I20260929 00:59:17.121331 13641 consensus_queue.cc:260] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.121464 13642 raft_consensus.cc:348] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.121776 13642 raft_consensus.cc:374] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:17.121855 13642 raft_consensus.cc:729] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda39cc835844a4599a57d6399d22c39, State: Initialized, Role: FOLLOWER
I20260929 00:59:17.122339 13642 consensus_queue.cc:260] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.122525 13641 ts_tablet_manager.cc:1428] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 00:59:17.122946 13340 heartbeater.cc:503] Master 127.0.98.62:36151 was elected leader, sending a full tablet report...
I20260929 00:59:17.123289 13642 ts_tablet_manager.cc:1428] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:59:17.123589 13475 heartbeater.cc:503] Master 127.0.98.62:36151 was elected leader, sending a full tablet report...
I20260929 00:59:17.129987 13644 tablet_bootstrap.cc:492] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: No bootstrap required, opened a new log
I20260929 00:59:17.130318 13644 ts_tablet_manager.cc:1397] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260929 00:59:17.135142 13644 raft_consensus.cc:348] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.135516 13644 raft_consensus.cc:374] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:17.135639 13644 raft_consensus.cc:729] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5538978c764c51ba60d723e9d919cd, State: Initialized, Role: FOLLOWER
I20260929 00:59:17.136262 13644 consensus_queue.cc:260] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.136556 13649 raft_consensus.cc:482] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:17.136742 13649 raft_consensus.cc:504] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.137394 13644 ts_tablet_manager.cc:1428] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 00:59:17.137794 13608 heartbeater.cc:503] Master 127.0.98.62:36151 was elected leader, sending a full tablet report...
I20260929 00:59:17.139694 13649 leader_election.cc:288] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fda39cc835844a4599a57d6399d22c39 (127.0.98.2:38451), f1503504d164461aa2085289744eafbd (127.0.98.1:46681)
I20260929 00:59:17.144300 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.138095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38451 (local address 127.0.98.3:41657)
0929 00:59:17.139998 (+ 1903us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:17.140012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:17.140154 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:17.140493 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:17.140495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:17.140510 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:17.140648 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.140654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.141660 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.141663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.142882 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.142893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.143873 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.143876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.143974 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.143997 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:59:17.144077 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 00:59:17.144148 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1844,"thread_start_us":412,"threads_started":1}
I20260929 00:59:17.144341 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.138359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41657 (local address 127.0.98.2:38451)
0929 00:59:17.138474 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.138478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:17.140084 (+ 1606us) server_negotiation.cc:407] Connection header received
0929 00:59:17.140266 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:17.140269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:17.140316 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:17.140359 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:17.140763 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.141512 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.143049 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.143638 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.143716 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.143778 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:17.144122 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 00:59:17.144213 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:59:17.145550 13429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8fe18d216048caa3e1b8dc731b6721" candidate_uuid: "5d5538978c764c51ba60d723e9d919cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda39cc835844a4599a57d6399d22c39" is_pre_election: true
I20260929 00:59:17.145881 13429 raft_consensus.cc:2454] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d5538978c764c51ba60d723e9d919cd in term 0.
I20260929 00:59:17.146481 13497 leader_election.cc:302] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d5538978c764c51ba60d723e9d919cd, fda39cc835844a4599a57d6399d22c39; no voters:
I20260929 00:59:17.150254 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.139540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58159 (local address 127.0.98.1:46681)
0929 00:59:17.139662 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.139666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:17.144886 (+ 5220us) server_negotiation.cc:407] Connection header received
0929 00:59:17.145010 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:17.145014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:17.145078 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:17.145165 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:17.146563 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.147351 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.148877 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.149509 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.149593 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.149658 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:17.149994 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 00:59:17.150085 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:59:17.150322 13650 raft_consensus.cc:2781] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:17.150418 13650 raft_consensus.cc:482] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:17.150492 13650 raft_consensus.cc:3037] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:17.150830 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.139211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46681 (local address 127.0.98.3:58159)
0929 00:59:17.144754 (+ 5543us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:17.144769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:17.144886 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:17.146283 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:17.146286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:17.146306 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:17.146423 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.146429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.147512 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.147515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.148733 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:17.148744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.149749 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.149753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:17.149853 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.149876 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:59:17.150609 (+ 733us) client_negotiation.cc:240] Negotiation successful
0929 00:59:17.150686 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5480,"thread_start_us":99,"threads_started":1}
I20260929 00:59:17.151751 13294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8fe18d216048caa3e1b8dc731b6721" candidate_uuid: "5d5538978c764c51ba60d723e9d919cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1503504d164461aa2085289744eafbd" is_pre_election: true
I20260929 00:59:17.152105 13294 raft_consensus.cc:2454] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d5538978c764c51ba60d723e9d919cd in term 0.
I20260929 00:59:17.152418 13650 raft_consensus.cc:504] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.153128 13429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8fe18d216048caa3e1b8dc731b6721" candidate_uuid: "5d5538978c764c51ba60d723e9d919cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda39cc835844a4599a57d6399d22c39"
I20260929 00:59:17.153260 13429 raft_consensus.cc:3037] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:17.153420 13294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8fe18d216048caa3e1b8dc731b6721" candidate_uuid: "5d5538978c764c51ba60d723e9d919cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1503504d164461aa2085289744eafbd"
I20260929 00:59:17.153525 13294 raft_consensus.cc:3037] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:17.154664 13429 raft_consensus.cc:2454] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d5538978c764c51ba60d723e9d919cd in term 1.
I20260929 00:59:17.154757 13294 raft_consensus.cc:2454] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d5538978c764c51ba60d723e9d919cd in term 1.
I20260929 00:59:17.154995 13497 leader_election.cc:302] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d5538978c764c51ba60d723e9d919cd, fda39cc835844a4599a57d6399d22c39; no voters:
I20260929 00:59:17.155166 13649 raft_consensus.cc:2781] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:17.155422 13649 raft_consensus.cc:686] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 1 LEADER]: Becoming Leader. State: Replica: 5d5538978c764c51ba60d723e9d919cd, State: Running, Role: LEADER
I20260929 00:59:17.155781 13649 consensus_queue.cc:237] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260929 00:59:17.159534 13141 catalog_manager.cc:5495] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d5538978c764c51ba60d723e9d919cd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5d5538978c764c51ba60d723e9d919cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5538978c764c51ba60d723e9d919cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } health_report { overall_health: UNKNOWN } } }
I20260929 00:59:17.160677 13650 leader_election.cc:288] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [CANDIDATE]: Term 1 election: Requested vote from peers fda39cc835844a4599a57d6399d22c39 (127.0.98.2:38451), f1503504d164461aa2085289744eafbd (127.0.98.1:46681)
I20260929 00:59:17.172811 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.168028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.0.98.3:34335)
0929 00:59:17.168176 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.168181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:17.168266 (+ 85us) server_negotiation.cc:407] Connection header received
0929 00:59:17.168423 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:17.168427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:17.168504 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:17.168626 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:17.168879 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.169663 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.170837 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:17.171183 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:17.171297 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:17.171615 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:17.172172 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:17.172293 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:17.172606 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 00:59:17.172667 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:59:17.179786 13294 raft_consensus.cc:1261] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd [term 1 FOLLOWER]: Refusing update from remote peer 5d5538978c764c51ba60d723e9d919cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:17.179785 13429 raft_consensus.cc:1261] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 1 FOLLOWER]: Refusing update from remote peer 5d5538978c764c51ba60d723e9d919cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:17.180536 13650 consensus_queue.cc:1035] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1503504d164461aa2085289744eafbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:17.180683 13649 consensus_queue.cc:1035] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda39cc835844a4599a57d6399d22c39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:17.191246 13656 mvcc.cc:205] Tried to move back new op lower bound from 7334476010199928832 to 7334476010113601536. Current Snapshot: MvccSnapshot[applied={T|T < 7334476010199928832}]
I20260929 00:59:17.192915 13657 mvcc.cc:205] Tried to move back new op lower bound from 7334476010199928832 to 7334476010113601536. Current Snapshot: MvccSnapshot[applied={T|T < 7334476010199928832}]
W20260929 00:59:17.193545 13341 tablet.cc:2369] T 3d8fe18d216048caa3e1b8dc731b6721 P f1503504d164461aa2085289744eafbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:59:17.194506 13658 mvcc.cc:205] Tried to move back new op lower bound from 7334476010199928832 to 7334476010113601536. Current Snapshot: MvccSnapshot[applied={T|T < 7334476010199928832}]
I20260929 00:59:17.203459 13142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:17.203655 13142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:17.206637 13142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0314485b7a34b518402edeecece7770: Sending DeleteTablet for 3 replicas of tablet 3d8fe18d216048caa3e1b8dc731b6721
I20260929 00:59:17.207953 13409 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8fe18d216048caa3e1b8dc731b6721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42940
I20260929 00:59:17.207954 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8fe18d216048caa3e1b8dc731b6721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37540
I20260929 00:59:17.209030 13542 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8fe18d216048caa3e1b8dc731b6721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55206
I20260929 00:59:17.210232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 13180
I20260929 00:59:17.214020 13665 tablet_replica.cc:330] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: stopping tablet replica
I20260929 00:59:17.214301 13667 tablet_replica.cc:330] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: stopping tablet replica
I20260929 00:59:17.214757 13667 raft_consensus.cc:2229] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:59:17.215139 13665 raft_consensus.cc:2229] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:17.215579 13665 raft_consensus.cc:2258] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:17.216043 13667 raft_consensus.cc:2258] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:17.218232 13665 ts_tablet_manager.cc:1905] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:17.219604 13667 ts_tablet_manager.cc:1905] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:17.223397 13665 ts_tablet_manager.cc:1918] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:17.225657 13667 ts_tablet_manager.cc:1918] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:17.225821 13667 log.cc:1196] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/3d8fe18d216048caa3e1b8dc731b6721
I20260929 00:59:17.226253 13667 ts_tablet_manager.cc:1939] T 3d8fe18d216048caa3e1b8dc731b6721 P 5d5538978c764c51ba60d723e9d919cd: Deleting consensus metadata
I20260929 00:59:17.226419 13665 log.cc:1196] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/3d8fe18d216048caa3e1b8dc731b6721
W20260929 00:59:17.226608 13129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46681 (error 108)
I20260929 00:59:17.227996 13126 catalog_manager.cc:4841] TS 5d5538978c764c51ba60d723e9d919cd (127.0.98.3:34335): tablet 3d8fe18d216048caa3e1b8dc731b6721 (table test-table [id=ed5e19f48a76468bb2a8f65a68e8f264]) successfully deleted
I20260929 00:59:17.228454 13665 ts_tablet_manager.cc:1939] T 3d8fe18d216048caa3e1b8dc731b6721 P fda39cc835844a4599a57d6399d22c39: Deleting consensus metadata
I20260929 00:59:17.229756 13128 catalog_manager.cc:4841] TS fda39cc835844a4599a57d6399d22c39 (127.0.98.2:38451): tablet 3d8fe18d216048caa3e1b8dc731b6721 (table test-table [id=ed5e19f48a76468bb2a8f65a68e8f264]) successfully deleted
I20260929 00:59:17.230355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 13344
I20260929 00:59:17.231315 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.230598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46681 (local address 127.0.0.1:37544)
0929 00:59:17.231128 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:17.231209 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420}
W20260929 00:59:17.232451 13129 catalog_manager.cc:4584] TS f1503504d164461aa2085289744eafbd (127.0.98.1:46681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d8fe18d216048caa3e1b8dc731b6721: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46681: connect: Connection refused (error 111)
I20260929 00:59:17.238602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 13478
I20260929 00:59:17.246091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 13109
I20260929 00:59:17.253569 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 00:59:17.587868 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 00:59:17.587967 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 00:59:17.932934 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 00:59:17.933 UTC [12176] LOG: received smart shutdown request
2026-09-29 00:59:17.937 UTC [12176] LOG: background worker "logical replication launcher" (PID 12185) exited with exit code 1
2026-09-29 00:59:17.937 UTC [12180] LOG: shutting down
2026-09-29 00:59:18.103 UTC [12176] LOG: database system is shut down
2026-09-29T00:59:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47119 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:18 dist-test-slave-30s6 krb5kdc[13677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:18 dist-test-slave-30s6 krb5kdc[13677](info): set up 2 sockets
Sep 29 00:59:18 dist-test-slave-30s6 krb5kdc[13677](info): commencing operation
krb5kdc: starting...
W20260929 00:59:19.286294 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:59:19 dist-test-slave-30s6 krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:59:19.405495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46077
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46077,127.0.98.61:37847,127.0.98.60:37777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:19.515771 13690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:19.516094 13690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:19.516192 13690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:19.520329 13690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:19.520416 13690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:19.520454 13690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:19.520509 13690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:19.520545 13690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:19.525756 13690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46077,127.0.98.61:37847,127.0.98.60:37777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:19.526917 13690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:19.527948 13690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:19.536274 13695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:19.536244 13698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:19.536306 13696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:19.536567 13690 server_base.cc:1034] running on GCE node
I20260929 00:59:19.537153 13690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:19.538277 13690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:19.539463 13690 hybrid_clock.cc:648] HybridClock initialized: now 1790643559539450 us; error 37 us; skew 500 ppm
Sep 29 00:59:19 dist-test-slave-30s6 krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:19.542325 13690 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:59:19.543228 13690 webserver.cc:460] Webserver started at http://127.0.98.62:41473/ using document root <none> and password file <none>
I20260929 00:59:19.543780 13690 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:19.543870 13690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:19.544094 13690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:19.546052 13690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "20053fe7d4bc4c81b2c80a43ff9a5199"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "7829afb15e9ae448ee487a2489c1842d"
server_key_iv: "9a4781c07f08f087878c0aa5ae3d56c9"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.546664 13690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "20053fe7d4bc4c81b2c80a43ff9a5199"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "7829afb15e9ae448ee487a2489c1842d"
server_key_iv: "9a4781c07f08f087878c0aa5ae3d56c9"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.550161 13690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:19.552532 13705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:19.553723 13690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 00:59:19.553861 13690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "20053fe7d4bc4c81b2c80a43ff9a5199"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "7829afb15e9ae448ee487a2489c1842d"
server_key_iv: "9a4781c07f08f087878c0aa5ae3d56c9"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.553967 13690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:19.572396 13690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:19.575351 13690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:19.575556 13690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:19.582757 13690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46077
I20260929 00:59:19.582782 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46077 every 8 connection(s)
I20260929 00:59:19.583678 13690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:19.586380 13758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:19.590040 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:19.591837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 13690
I20260929 00:59:19.591992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:59:19.592271 392 external_mini_cluster.cc:1428] Setting key 5203859b74b0ce62c462500ea3ebae07
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 29 00:59:19 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260929 00:59:19.605898 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.592348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.0.98.62:46077)
0929 00:59:19.592909 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:19.592923 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:19.592969 (+ 46us) server_negotiation.cc:407] Connection header received
0929 00:59:19.594124 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:19.594129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:19.594361 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:19.594514 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:19.595633 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.596588 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.597631 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.598318 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.598509 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.600840 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:19.600845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:19.600856 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:19.600877 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:19.602669 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.603213 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.603218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.603219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.603327 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.603680 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.603682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.603683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.604055 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:19.604194 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:19.604561 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 00:59:19.604838 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":423,"spinlock_wait_cycles":9344,"thread_start_us":148,"threads_started":1}
I20260929 00:59:19.605898 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.592349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46077 (local address 127.0.0.1:51700)
0929 00:59:19.592814 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.592867 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:19.593799 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:19.594561 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:19.594567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:19.595072 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:19.595472 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:19.595493 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.596722 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.596729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:19.597504 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:19.597514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.598473 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.598476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:19.598567 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.599206 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:19.599240 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:19.600696 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:19.602773 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:19.602781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:19.602789 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:19.603097 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:19.603430 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:19.603437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:19.603442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:19.603591 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:19.604183 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:19.604189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:19.604324 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 00:59:19.604493 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 00:59:19.604884 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":111,"spinlock_wait_cycles":896,"thread_start_us":128,"threads_started":1}
I20260929 00:59:19.608469 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:19.609248 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.609003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39826)
0929 00:59:19.609122 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.609189 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:59:19.609483 13708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
I20260929 00:59:19.610378 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.610150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39838)
0929 00:59:19.610281 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.610330 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 00:59:19.610512 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37847: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 00:59:19.618537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37847
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46077,127.0.98.61:37847,127.0.98.60:37777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:19.659749 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } attempt: 1
I20260929 00:59:19.660630 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.660309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39848)
0929 00:59:19.660464 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.660541 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 00:59:19.661423 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.661189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39864)
0929 00:59:19.661306 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.661363 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 00:59:19.661589 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37847: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
W20260929 00:59:19.726014 13765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:19.726346 13765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:19.726429 13765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 00:59:19.729782 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } attempt: 2
W20260929 00:59:19.730515 13765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:19.730585 13765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:19.730628 13765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:19.730684 13765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:19.730724 13765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:19.730726 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.730366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39880)
0929 00:59:19.730517 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.730621 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 00:59:19.731493 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.731251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39890)
0929 00:59:19.731387 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.731440 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 00:59:19.731653 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37847: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37847: connect: Connection refused (error 111)
I20260929 00:59:19.736352 13765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46077,127.0.98.61:37847,127.0.98.60:37777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:19.737432 13765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:19.738504 13765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:19.747380 13771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:19.747345 13770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:19.747403 13765 server_base.cc:1034] running on GCE node
W20260929 00:59:19.747355 13773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:19.748083 13765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:19.749138 13765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:19.750321 13765 hybrid_clock.cc:648] HybridClock initialized: now 1790643559750310 us; error 29 us; skew 500 ppm
Sep 29 00:59:19 dist-test-slave-30s6 krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:19.753046 13765 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 00:59:19.753921 13765 webserver.cc:460] Webserver started at http://127.0.98.61:37767/ using document root <none> and password file <none>
I20260929 00:59:19.754516 13765 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:19.754606 13765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:19.754832 13765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:19.756819 13765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/data/instance:
uuid: "ee14b5d989e046c3a9e4f8de3df5aabb"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "a7248f95fd22f057bf111f830cb3d0d7"
server_key_iv: "40e3d288cbfb3006d924b8e8863dc0a0"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.757373 13765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/wal/instance:
uuid: "ee14b5d989e046c3a9e4f8de3df5aabb"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "a7248f95fd22f057bf111f830cb3d0d7"
server_key_iv: "40e3d288cbfb3006d924b8e8863dc0a0"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.761041 13765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 00:59:19.763365 13780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:19.764529 13765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:59:19.764667 13765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/wal
uuid: "ee14b5d989e046c3a9e4f8de3df5aabb"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "a7248f95fd22f057bf111f830cb3d0d7"
server_key_iv: "40e3d288cbfb3006d924b8e8863dc0a0"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.764773 13765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:19.785153 13765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:19.788621 13765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:19.788843 13765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:19.795943 13765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37847
I20260929 00:59:19.795994 13832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37847 every 8 connection(s)
I20260929 00:59:19.796921 13765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
I20260929 00:59:19.799742 13833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:19.803556 13833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:19.804183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 13765
I20260929 00:59:19.804311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-1/wal/instance
I20260929 00:59:19.804577 392 external_mini_cluster.cc:1428] Setting key 8d0ea5bfd708da7d953b35a92699fafd
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 00:59:19 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:19.820729 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.805686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.0.98.62:46077)
0929 00:59:19.805827 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:19.805831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:19.806839 (+ 1008us) server_negotiation.cc:407] Connection header received
0929 00:59:19.807786 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:19.807793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:19.807869 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:19.807982 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:19.809249 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.810176 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.811514 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.811924 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.812071 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.816526 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:19.816530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:19.816532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:19.816560 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:19.818341 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.818858 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.818864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.818882 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.818982 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.819334 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.819340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.819345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.819641 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:19.819759 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:19.820299 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 00:59:19.820468 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:59:19.821345 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.806120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46077 (local address 127.0.0.1:51714)
0929 00:59:19.806660 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.806712 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:19.807560 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:19.808142 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:19.808148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:19.808576 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:19.809074 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:19.809087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.810338 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.810341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:19.811338 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:19.811347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.812087 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.812090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:19.812232 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.812866 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:19.812906 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:19.816333 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:19.818471 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:19.818477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:19.818490 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:19.818750 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:19.819085 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:19.819087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:19.819089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:19.819229 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:19.819758 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:19.819764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:19.820022 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 00:59:19.820187 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 00:59:19.820463 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
I20260929 00:59:19.822827 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } attempt: 3
I20260929 00:59:19.822918 13833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:19 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 00:59:19 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 00:59:19.833034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37777
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46077,127.0.98.61:37847,127.0.98.60:37777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:19.834797 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.823444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39912)
0929 00:59:19.823703 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.823714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:19.823782 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:19.824334 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:19.824336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:19.824495 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:19.824575 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:19.824580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.825979 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.825982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:19.826799 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:19.826822 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.827864 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.827868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:19.828016 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.828532 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:19.828567 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:19.830591 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:19.833175 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:19.833178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:19.833180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:19.833426 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:19.833770 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:19.833776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:19.833781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:19.833897 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:19.834438 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:19.834441 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:19.834516 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 00:59:19.834576 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 00:59:19.834675 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":23,"spinlock_wait_cycles":39168}
I20260929 00:59:19.835048 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.823258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39904)
0929 00:59:19.823445 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.823460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:19.823537 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:19.824199 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:19.824202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:19.824372 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:19.824972 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:19.824982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.825909 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.825914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:19.826799 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:19.826809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.827864 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.827868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:19.827976 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.828603 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:19.828621 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:19.830047 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:19.833185 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:19.833188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:19.833190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:19.833423 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:19.833784 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:19.833788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:19.833790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:19.833878 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:19.834459 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:19.834465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:19.834579 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 00:59:19.834670 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 00:59:19.834822 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 00:59:19.835561 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.823720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.0.98.61:37847)
0929 00:59:19.824003 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:19.824011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:19.824021 (+ 10us) server_negotiation.cc:407] Connection header received
0929 00:59:19.824048 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:19.824050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:19.824078 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:19.824202 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:19.824735 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.825818 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.826925 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.827734 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.827947 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.830744 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:19.830747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:19.830748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:19.830766 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:19.833062 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.833551 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.833557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.833559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.833657 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.834010 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.834016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.834021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.834315 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:19.835173 (+ 858us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:19.835284 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 00:59:19.835436 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":220,"server-negotiator.queue_time_us":229,"spinlock_wait_cycles":15616,"thread_start_us":125,"threads_started":1}
I20260929 00:59:19.835757 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.823323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.0.98.61:37847)
0929 00:59:19.823764 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:19.823775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:19.823795 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:59:19.823852 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:19.823858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:19.824061 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:19.824202 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:19.825089 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.825796 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.826925 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.827734 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.827951 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.830284 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:19.830297 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:19.830317 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:19.830348 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:19.833077 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.833551 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.833557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.833558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.833672 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.833979 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.833982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.833984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.834317 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:19.835363 (+ 1046us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:19.835451 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 00:59:19.835574 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":391,"server-negotiator.queue_time_us":350,"spinlock_wait_cycles":6144,"thread_start_us":99,"threads_started":1}
I20260929 00:59:19.837459 13833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:19.837580 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:19.838294 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.837927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49162)
0929 00:59:19.838074 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.838171 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 00:59:19.838354 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.838062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49166)
0929 00:59:19.838154 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.838270 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 00:59:19.838518 13782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
I20260929 00:59:19.838990 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.838722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49172)
0929 00:59:19.838848 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.838927 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 00:59:19.839232 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
I20260929 00:59:19.839605 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.839306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49178)
0929 00:59:19.839473 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.839530 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 00:59:19.839787 13833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
I20260929 00:59:19.870406 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } attempt: 1
I20260929 00:59:19.871160 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.870871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49190)
0929 00:59:19.871020 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.871086 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 00:59:19.871841 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.871632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49194)
0929 00:59:19.871743 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.871793 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 00:59:19.871987 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
I20260929 00:59:19.889035 13833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } attempt: 1
I20260929 00:59:19.889881 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.889529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49198)
0929 00:59:19.889702 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.889776 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 00:59:19.890726 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.890487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49202)
0929 00:59:19.890606 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.890661 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:59:19.890866 13833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
W20260929 00:59:19.942224 13840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:19.942471 13840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:19.942510 13840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:19.946395 13840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:19.946444 13840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:19.946463 13840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:19.946480 13840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:19.946496 13840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:19.947188 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } attempt: 2
I20260929 00:59:19.948047 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.947733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49210)
0929 00:59:19.947897 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.947970 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 00:59:19.948870 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.948664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49218)
0929 00:59:19.948768 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.948818 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 00:59:19.949079 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
I20260929 00:59:19.952056 13840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46077,127.0.98.61:37847,127.0.98.60:37777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:19.953040 13840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:19.954017 13840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:59:19.959018 13833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } attempt: 2
I20260929 00:59:19.959704 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.959438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49234)
0929 00:59:19.959571 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.959631 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 00:59:19.960374 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.960167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49242)
0929 00:59:19.960270 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:19.960319 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 00:59:19.960498 13833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37777: connect: Connection refused (error 111)
W20260929 00:59:19.962628 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:19.962670 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:19.962670 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:19.962901 13840 server_base.cc:1034] running on GCE node
I20260929 00:59:19.963527 13840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:19.964556 13840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:19.965744 13840 hybrid_clock.cc:648] HybridClock initialized: now 1790643559965719 us; error 48 us; skew 500 ppm
Sep 29 00:59:19 dist-test-slave-30s6 krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:19.968582 13840 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 00:59:19.969451 13840 webserver.cc:460] Webserver started at http://127.0.98.60:40375/ using document root <none> and password file <none>
I20260929 00:59:19.969954 13840 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:19.970010 13840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:19.970163 13840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:19.972090 13840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/data/instance:
uuid: "9de30442cd1642e9a2b3cd5be7c8d27d"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "a7f2af39fb1b60f8be4c1ccf7c42f0f7"
server_key_iv: "60ca03c0aab4c66d77b9fb0786fa45bc"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.972587 13840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/wal/instance:
uuid: "9de30442cd1642e9a2b3cd5be7c8d27d"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "a7f2af39fb1b60f8be4c1ccf7c42f0f7"
server_key_iv: "60ca03c0aab4c66d77b9fb0786fa45bc"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.976001 13840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:59:19.978250 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:19.979475 13840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:19.979597 13840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/wal
uuid: "9de30442cd1642e9a2b3cd5be7c8d27d"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-30s6"
server_key: "a7f2af39fb1b60f8be4c1ccf7c42f0f7"
server_key_iv: "60ca03c0aab4c66d77b9fb0786fa45bc"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.979701 13840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:19.992477 13840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:20.002377 13840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:20.002593 13840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:20.009495 13840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37777
I20260929 00:59:20.009513 13908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37777 every 8 connection(s)
I20260929 00:59:20.010456 13840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
I20260929 00:59:20.013154 13909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:20.017362 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:20.019629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 13840
I20260929 00:59:20.019753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643398448434-392-0/minicluster-data/master-2/wal/instance
I20260929 00:59:20.020010 392 external_mini_cluster.cc:1428] Setting key 8dd88513d1314ad2946636e55668dadd
Sep 29 00:59:20 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 00:59:20 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 00:59:20.031206 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.019698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.0.98.62:46077)
0929 00:59:20.019840 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.019845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.020831 (+ 986us) server_negotiation.cc:407] Connection header received
0929 00:59:20.021795 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.021799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.021855 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.021956 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.022801 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.023309 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.024271 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.024508 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.024621 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.027170 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.027174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.027179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.027212 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.028816 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.029377 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.029381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.029383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.029506 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.029854 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.029856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.029858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.030083 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.030217 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.030761 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.030888 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:59:20.032033 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.020651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.0.98.60:37777)
0929 00:59:20.021034 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.021046 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.021320 (+ 274us) server_negotiation.cc:407] Connection header received
0929 00:59:20.021592 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.021606 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.021784 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.021976 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.022801 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.023625 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.024450 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.024793 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.024917 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.027613 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.027622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.027638 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.027666 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.029371 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.029813 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.029822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.029825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.029919 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.030203 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.030206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.030207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.030502 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.030705 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.030926 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.031084 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":300,"thread_start_us":129,"threads_started":1}
I20260929 00:59:20.032017 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.020137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46077 (local address 127.0.0.1:51718)
0929 00:59:20.020665 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:20.020728 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:20.021591 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:20.022011 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:20.022016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:20.022421 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:20.022660 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.022672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.023417 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.023424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.024145 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.024153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.024652 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.024656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.024819 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.025425 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:20.025448 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:20.027020 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:20.028929 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.028947 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.028956 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.029274 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.029602 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.029604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.029606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.029732 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.030209 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:20.030215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:20.030520 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 00:59:20.030688 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 00:59:20.031033 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"spinlock_wait_cycles":3840,"thread_start_us":129,"threads_started":1}
I20260929 00:59:20.033279 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:20 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 00:59:20 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:20.041498 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.033757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37847 (local address 127.0.0.1:39918)
0929 00:59:20.033878 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:20.033893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:20.033967 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:20.034267 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:20.034269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:20.034472 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:20.034587 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.034594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.035334 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.035340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.035845 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.035853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.036318 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.036322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.036416 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.036883 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:20.036896 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:20.038110 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:20.039970 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.039974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.039976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.040255 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.040536 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.040539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.040541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.040623 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.040997 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:20.041000 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:20.041097 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 00:59:20.041199 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 00:59:20.041324 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:59:20.041532 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.033870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.0.98.61:37847)
0929 00:59:20.034018 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.034025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.034038 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:59:20.034078 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.034082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.034126 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.034219 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.034702 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.035221 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.035972 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.036207 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.036321 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.038276 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.038282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.038284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.038315 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.039860 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.040351 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.040356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.040358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.040434 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.040712 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.040717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.040720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.040892 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.041007 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.041261 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.041374 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:59:20.042330 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:20.042963 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.034575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.0.98.62:46077)
0929 00:59:20.034685 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.034690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.034701 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:59:20.034793 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.034798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.034833 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.034925 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.035481 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.035946 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.036763 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.037101 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.037211 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.039747 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.039750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.039752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.039772 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.041202 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.041656 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.041660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.041664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.041761 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.042042 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.042045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.042047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.042291 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.042497 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.042693 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.042820 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 00:59:20 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 00:59:20.048259 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } attempt: 3
Sep 29 00:59:20 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 00:59:20.051055 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.042830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49256)
0929 00:59:20.043024 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:20.043040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:20.043143 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:20.043414 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:20.043416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:20.043561 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:20.043666 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.043672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.044600 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.044602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.045176 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.045185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.045619 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.045623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.045719 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.046156 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:20.046167 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:20.047548 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:20.049406 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.049409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.049411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.049666 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.049966 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.049969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.049971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.050059 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.050563 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:20.050565 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:20.050652 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 00:59:20.050740 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:59:20.050847 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 00:59:20.051659 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.042916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.0.98.60:37777)
0929 00:59:20.043011 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.043016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.043091 (+ 75us) server_negotiation.cc:407] Connection header received
0929 00:59:20.043252 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.043256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.043299 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.043379 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.043759 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.044496 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.045284 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.045515 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.045593 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.047679 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.047682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.047684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.047705 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.049296 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.049770 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.049773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.049774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.049858 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.050161 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.050164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.050166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.050451 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.051307 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.051378 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.051488 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 00:59:20.051648 13833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } attempt: 3
I20260929 00:59:20.053836 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.044151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.98.61:37847)
0929 00:59:20.044243 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.044247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.044260 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:59:20.044318 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.044320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.044350 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.044409 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.045012 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.045618 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.046644 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.046924 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.047004 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.049864 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.049868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.049870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.049893 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.051681 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.052163 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.052166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.052169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.052252 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.052557 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.052560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.052562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.052786 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.052980 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.053593 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.053697 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 29 00:59:20 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 00:59:20.056197 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d: Bootstrap starting.
Sep 29 00:59:20 dist-test-slave-30s6 krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 00:59:20.058616 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.048683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49266)
0929 00:59:20.048833 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:20.048850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:20.048927 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:20.049525 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:20.049529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:20.049689 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:20.049801 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.049812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.051282 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.051286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.051982 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.051992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.052596 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.052600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.052677 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.053098 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:20.053112 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:20.054582 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:20.056813 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.056816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.056818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.057104 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.057443 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.057446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.057448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.057556 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.058027 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:20.058030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:20.058130 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 00:59:20.058283 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 00:59:20.058423 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:59:20.058772 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:20.059266 13909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:20.059491 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.048909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.0.98.60:37777)
0929 00:59:20.049295 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.049307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.049321 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:59:20.049356 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.049360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.049415 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.049492 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.049950 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.051148 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.052126 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.052466 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.052887 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.054718 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.054721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.054723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.054746 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.056687 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.057246 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.057251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.057253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.057335 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.057679 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.057682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.057684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.057939 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.058036 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.059101 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.059229 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"spinlock_wait_cycles":4096,"thread_start_us":149,"threads_started":1}
I20260929 00:59:20.060346 13909 log.cc:824] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:20.061107 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.051982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37777 (local address 127.0.0.1:49280)
0929 00:59:20.052342 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:20.052356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:20.052449 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:20.052706 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:20.052709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:20.052885 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:20.052991 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.053001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.053982 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.053986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.054822 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:20.054832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.055722 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.055726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:20.055811 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.056177 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:20.056194 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:20.057610 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:20.059619 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.059622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.059624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.059864 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.060166 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:20.060169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:20.060171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:20.060258 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:20.060690 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:20.060693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:20.060773 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 00:59:20.060846 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 00:59:20.060961 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288}
I20260929 00:59:20.061594 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.052047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.0.98.60:37777)
0929 00:59:20.052320 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.052326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.052403 (+ 77us) server_negotiation.cc:407] Connection header received
0929 00:59:20.052538 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.052543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.052585 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.053099 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.053177 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.053834 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.055236 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.055576 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.055718 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.057738 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.057742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.057744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.057773 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.059491 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.059978 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.059980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.059983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.060062 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.060372 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.060377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.060382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.060580 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.061276 (+ 696us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.061346 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.061451 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"spinlock_wait_cycles":2304}
I20260929 00:59:20.062278 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: Bootstrap starting.
I20260929 00:59:20.064684 13758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:20.065016 13833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb: Bootstrap starting.
I20260929 00:59:20.065500 13758 log.cc:824] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:20.067268 13833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:20.067488 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d: No bootstrap required, opened a new log
I20260929 00:59:20.067485 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: No bootstrap required, opened a new log
I20260929 00:59:20.068187 13833 log.cc:824] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:20.070132 13833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb: No bootstrap required, opened a new log
I20260929 00:59:20.071487 13758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.071499 13909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.071700 13909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:20.071700 13758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:20.071748 13909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de30442cd1642e9a2b3cd5be7c8d27d, State: Initialized, Role: FOLLOWER
I20260929 00:59:20.071756 13758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20053fe7d4bc4c81b2c80a43ff9a5199, State: Initialized, Role: FOLLOWER
I20260929 00:59:20.072201 13758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.072201 13909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.073436 13758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:20.073436 13909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:20.074842 13833 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.075125 13833 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:20.075229 13833 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee14b5d989e046c3a9e4f8de3df5aabb, State: Initialized, Role: FOLLOWER
I20260929 00:59:20.075579 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.062681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.0.98.61:37847)
0929 00:59:20.066488 (+ 3807us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.066492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.066504 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:59:20.066535 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.066538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.066574 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.066647 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.067556 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.068268 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.069612 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.069904 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.070015 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.071766 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.071769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.071772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.071794 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.073459 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.073938 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.073942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.073943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.074024 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.074361 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.074364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.074365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.074565 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.074679 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.075324 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.075429 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3729}
W20260929 00:59:20.077001 13711 tablet.cc:2369] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:59:20.077219 13833 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.078574 13833 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:20.077670 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.079341 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:20.081447 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.078531 13690 master_runner.cc:186] Error getting master registration for 127.0.98.61:37847: OK, instance_id { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" instance_seqno: 1790643559793846 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:59:20.082118 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: This master's current role is: FOLLOWER
W20260929 00:59:20.083866 13960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:59:20.084000 13960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:59:20.084259 13690 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:59:20.082676 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.063028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.0.98.62:46077)
0929 00:59:20.063125 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.063129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.063703 (+ 574us) server_negotiation.cc:407] Connection header received
0929 00:59:20.063851 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.063854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.063895 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.063969 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.064763 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.065481 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.066644 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.066963 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.067039 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.078734 (+ 11695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.078737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.078740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.078764 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.080563 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.081094 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.081100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.081103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.081196 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.081533 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.081537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.081539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.081778 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.081912 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.082340 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.082474 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 00:59:20.085237 13947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:59:20.085325 13947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:59:20.085551 13840 master_runner.cc:263] No leader master found from master 9de30442cd1642e9a2b3cd5be7c8d27d
I20260929 00:59:20.085603 13840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:59:20.085544 13925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.085654 13925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:20.089041 13765 master_runner.cc:263] No leader master found from master ee14b5d989e046c3a9e4f8de3df5aabb
I20260929 00:59:20.089099 13765 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 00:59:20.090615 13948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:59:20.090735 13948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:59:20.092537 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:20.066293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.0.98.60:37777)
0929 00:59:20.066391 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:20.066395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:20.075817 (+ 9422us) server_negotiation.cc:407] Connection header received
0929 00:59:20.075994 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:20.075997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:20.076051 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:20.076097 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:20.085220 (+ 9123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.085923 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.086964 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:20.087262 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:20.087335 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:20.089030 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:20.089033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:20.089036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:20.089060 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:20.090712 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.091193 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.091197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.091199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.091281 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:20.091596 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:20.091600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:20.091602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:20.091810 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:20.091917 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:20.092260 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 00:59:20.092372 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:59:20.157991 13926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:20.158396 13926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.159535 13926 leader_election.cc:288] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20053fe7d4bc4c81b2c80a43ff9a5199 (127.0.98.62:46077), ee14b5d989e046c3a9e4f8de3df5aabb (127.0.98.61:37847)
I20260929 00:59:20.160192 13806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" is_pre_election: true
I20260929 00:59:20.160539 13806 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9de30442cd1642e9a2b3cd5be7c8d27d in term 0.
I20260929 00:59:20.161155 13859 leader_election.cc:302] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9de30442cd1642e9a2b3cd5be7c8d27d, ee14b5d989e046c3a9e4f8de3df5aabb; no voters:
I20260929 00:59:20.161556 13926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:20.161638 13926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:20.161682 13926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:20.162880 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" is_pre_election: true
I20260929 00:59:20.163153 13732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9de30442cd1642e9a2b3cd5be7c8d27d in term 0.
I20260929 00:59:20.164638 13926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.165048 13926 leader_election.cc:288] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [CANDIDATE]: Term 1 election: Requested vote from peers 20053fe7d4bc4c81b2c80a43ff9a5199 (127.0.98.62:46077), ee14b5d989e046c3a9e4f8de3df5aabb (127.0.98.61:37847)
I20260929 00:59:20.165484 13806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb"
I20260929 00:59:20.165603 13806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:20.166971 13806 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9de30442cd1642e9a2b3cd5be7c8d27d in term 1.
I20260929 00:59:20.167778 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199"
I20260929 00:59:20.167896 13732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:20.169131 13732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9de30442cd1642e9a2b3cd5be7c8d27d in term 1.
I20260929 00:59:20.167371 13859 leader_election.cc:302] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9de30442cd1642e9a2b3cd5be7c8d27d, ee14b5d989e046c3a9e4f8de3df5aabb; no voters:
I20260929 00:59:20.170044 13926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:20.170333 13926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [term 1 LEADER]: Becoming Leader. State: Replica: 9de30442cd1642e9a2b3cd5be7c8d27d, State: Running, Role: LEADER
I20260929 00:59:20.170729 13926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } }
I20260929 00:59:20.176132 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de30442cd1642e9a2b3cd5be7c8d27d. Latest consensus state: current_term: 1 leader_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.176347 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:20.176792 13966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:20.177843 13966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:20.186028 13806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [term 1 FOLLOWER]: Refusing update from remote peer 9de30442cd1642e9a2b3cd5be7c8d27d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:20.186923 13964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:20.191320 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de30442cd1642e9a2b3cd5be7c8d27d. Latest consensus state: current_term: 1 leader_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.191452 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:20.195187 13732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [term 1 FOLLOWER]: Refusing update from remote peer 9de30442cd1642e9a2b3cd5be7c8d27d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:20.195855 13926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:20.199633 13925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de30442cd1642e9a2b3cd5be7c8d27d. Latest consensus state: current_term: 1 leader_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.199759 13925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:20.204254 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.204412 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:20.205786 13925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.205902 13925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:20.208797 13967 mvcc.cc:205] Tried to move back new op lower bound from 7334476022511357952 to 7334476022471520256. Current Snapshot: MvccSnapshot[applied={T|T < 7334476022511357952}]
I20260929 00:59:20.210618 13970 mvcc.cc:205] Tried to move back new op lower bound from 7334476022511357952 to 7334476022471520256. Current Snapshot: MvccSnapshot[applied={T|T < 7334476022511357952}]
I20260929 00:59:20.212556 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.212666 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:20.213572 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20053fe7d4bc4c81b2c80a43ff9a5199" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46077 } } peers { permanent_uuid: "ee14b5d989e046c3a9e4f8de3df5aabb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37847 } } peers { permanent_uuid: "9de30442cd1642e9a2b3cd5be7c8d27d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37777 } } }
I20260929 00:59:20.213682 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:20.215197 13968 mvcc.cc:205] Tried to move back new op lower bound from 7334476022511357952 to 7334476022471520256. Current Snapshot: MvccSnapshot[applied={T|T < 7334476022511357952}]
I20260929 00:59:20.218554 13966 catalog_manager.cc:1347] Generated new cluster ID: bbaed711a0494589a65ab58f6971b39a
I20260929 00:59:20.218611 13966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:20.238935 13966 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:20.242990 13966 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:20.257709 13966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9de30442cd1642e9a2b3cd5be7c8d27d: Generated new TSK 0
I20260929 00:59:20.261468 13966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:21.085301 13960 catalog_manager.cc:1259] Loaded cluster ID: bbaed711a0494589a65ab58f6971b39a
I20260929 00:59:21.085413 13960 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb: loading cluster ID for follower catalog manager: success
I20260929 00:59:21.087081 13960 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb: acquiring CA information for follower catalog manager: success
I20260929 00:59:21.088209 13960 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee14b5d989e046c3a9e4f8de3df5aabb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:59:21.091820 13948 catalog_manager.cc:1259] Loaded cluster ID: bbaed711a0494589a65ab58f6971b39a
I20260929 00:59:21.091886 13948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: loading cluster ID for follower catalog manager: success
I20260929 00:59:21.093447 13948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: acquiring CA information for follower catalog manager: success
I20260929 00:59:21.094590 13948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20053fe7d4bc4c81b2c80a43ff9a5199: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:59:21.335891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 13690
I20260929 00:59:21.342307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 13765
I20260929 00:59:21.347922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 13840
2026-09-29T00:59:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3128 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:21 dist-test-slave-30s6 krb5kdc[13981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:21 dist-test-slave-30s6 krb5kdc[13981](info): set up 2 sockets
Sep 29 00:59:21 dist-test-slave-30s6 krb5kdc[13981](info): commencing operation
krb5kdc: starting...
W20260929 00:59:22.386576 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:59:22 dist-test-slave-30s6 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:59:22.502166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42323
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:22.610147 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:22.610459 13994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:22.610535 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:22.614542 13994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:22.614598 13994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:22.614616 13994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:22.614669 13994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:22.614698 13994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:22.619971 13994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:22.620968 13994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:22.621948 13994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:22.630329 13999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:22.630416 13994 server_base.cc:1034] running on GCE node
W20260929 00:59:22.630329 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:22.630594 14002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:22.631214 13994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:22.632395 13994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:22.633594 13994 hybrid_clock.cc:648] HybridClock initialized: now 1790643562633577 us; error 27 us; skew 500 ppm
Sep 29 00:59:22 dist-test-slave-30s6 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:22.636322 13994 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:59:22.637164 13994 webserver.cc:460] Webserver started at http://127.0.98.62:33243/ using document root <none> and password file <none>
I20260929 00:59:22.637738 13994 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:22.637799 13994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:22.637967 13994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:22.639979 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "d8947b419f5d4a348bb9fc37ce9ba020"
format_stamp: "Formatted at 2026-09-29 00:59:22 on dist-test-slave-30s6"
server_key: "e776957f75d458ba4d3bb65c420693a1"
server_key_iv: "58ba196f318182328b6f8eeaa587b495"
server_key_version: "encryptionkey@0"
I20260929 00:59:22.640497 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8947b419f5d4a348bb9fc37ce9ba020"
format_stamp: "Formatted at 2026-09-29 00:59:22 on dist-test-slave-30s6"
server_key: "e776957f75d458ba4d3bb65c420693a1"
server_key_iv: "58ba196f318182328b6f8eeaa587b495"
server_key_version: "encryptionkey@0"
I20260929 00:59:22.644049 13994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:59:22.646324 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:22.647567 13994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:59:22.647696 13994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "d8947b419f5d4a348bb9fc37ce9ba020"
format_stamp: "Formatted at 2026-09-29 00:59:22 on dist-test-slave-30s6"
server_key: "e776957f75d458ba4d3bb65c420693a1"
server_key_iv: "58ba196f318182328b6f8eeaa587b495"
server_key_version: "encryptionkey@0"
I20260929 00:59:22.647805 13994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:22.671367 13994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:22.674744 13994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:22.674960 13994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:22.682024 13994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42323
I20260929 00:59:22.682055 14061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42323 every 8 connection(s)
I20260929 00:59:22.682946 13994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:22.685570 14062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:22.688088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 13994
I20260929 00:59:22.688182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:59:22.688457 392 external_mini_cluster.cc:1428] Setting key cd5cbf555ffe729067119c76682cb98b
I20260929 00:59:22.691708 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020: Bootstrap starting.
I20260929 00:59:22.694371 14062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:22.695168 14062 log.cc:824] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:22.697052 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020: No bootstrap required, opened a new log
Sep 29 00:59:22 dist-test-slave-30s6 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:22.701357 14062 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8947b419f5d4a348bb9fc37ce9ba020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42323 } }
I20260929 00:59:22.701627 14062 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:22.701685 14062 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8947b419f5d4a348bb9fc37ce9ba020, State: Initialized, Role: FOLLOWER
I20260929 00:59:22.702162 14062 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8947b419f5d4a348bb9fc37ce9ba020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42323 } }
I20260929 00:59:22.702340 14062 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:22.702422 14062 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:22.702524 14062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:22.703486 14062 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8947b419f5d4a348bb9fc37ce9ba020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42323 } }
I20260929 00:59:22.703828 14062 leader_election.cc:302] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8947b419f5d4a348bb9fc37ce9ba020; no voters:
I20260929 00:59:22.704175 14062 leader_election.cc:288] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:22.704202 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:22.689815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.98.62:42323)
0929 00:59:22.690285 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:22.690305 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:22.690349 (+ 44us) server_negotiation.cc:407] Connection header received
0929 00:59:22.691149 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:22.691172 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:22.691605 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:22.691965 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:22.692620 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:22.694089 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:22.695037 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:22.695697 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:22.695815 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:22.698553 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:22.698559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:22.698576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:22.698605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:22.701156 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:22.701561 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:22.701566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:22.701571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:22.701716 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:22.702035 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:22.702039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:22.702040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:22.702606 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:22.702802 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:22.703072 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 00:59:22.703358 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260929 00:59:22.704264 14067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:22.704448 14067 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [term 1 LEADER]: Becoming Leader. State: Replica: d8947b419f5d4a348bb9fc37ce9ba020, State: Running, Role: LEADER
I20260929 00:59:22.704769 14067 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8947b419f5d4a348bb9fc37ce9ba020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42323 } }
I20260929 00:59:22.705439 14062 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:22.705912 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8947b419f5d4a348bb9fc37ce9ba020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8947b419f5d4a348bb9fc37ce9ba020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42323 } } }
I20260929 00:59:22.706038 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:22.706288 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8947b419f5d4a348bb9fc37ce9ba020. Latest consensus state: current_term: 1 leader_uuid: "d8947b419f5d4a348bb9fc37ce9ba020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8947b419f5d4a348bb9fc37ce9ba020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42323 } } }
I20260929 00:59:22.706398 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:22.706790 14071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:22.707255 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:22.710386 14071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:22.715426 14071 catalog_manager.cc:1347] Generated new cluster ID: 33ebfc4fe0984ac897d10fd34380aa0a
I20260929 00:59:22.715490 14071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:22.735438 14071 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:22.736385 14071 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:22.742879 14071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8947b419f5d4a348bb9fc37ce9ba020: Generated new TSK 0
I20260929 00:59:22.743533 14071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:22.763271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 13994
2026-09-29T00:59:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1413 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:22 dist-test-slave-30s6 krb5kdc[14089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:22 dist-test-slave-30s6 krb5kdc[14089](info): set up 2 sockets
Sep 29 00:59:22 dist-test-slave-30s6 krb5kdc[14089](info): commencing operation
krb5kdc: starting...
W20260929 00:59:23.806600 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:59:23 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:59:23.950278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:24.067533 14102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:24.067871 14102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:24.067961 14102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:24.072017 14102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:24.072119 14102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:24.072158 14102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:24.072197 14102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:24.072227 14102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:24.077543 14102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:24.078648 14102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:24.079679 14102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:24.088522 14108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:24.088521 14107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:24.088554 14110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:24.088778 14102 server_base.cc:1034] running on GCE node
I20260929 00:59:24.089378 14102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:24.090551 14102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:24.091711 14102 hybrid_clock.cc:648] HybridClock initialized: now 1790643564091682 us; error 42 us; skew 500 ppm
Sep 29 00:59:24 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:24.094576 14102 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:59:24.095526 14102 webserver.cc:460] Webserver started at http://127.0.98.62:46045/ using document root <none> and password file <none>
I20260929 00:59:24.096148 14102 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:24.096246 14102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:24.096480 14102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:24.098616 14102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "0938835f982e4bff9bd647ed55764c80"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "99c67bc080307abb5134b0bfcecd358c"
server_key_iv: "52fbd9388d395d020f9f2645f3560af9"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.099195 14102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "0938835f982e4bff9bd647ed55764c80"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "99c67bc080307abb5134b0bfcecd358c"
server_key_iv: "52fbd9388d395d020f9f2645f3560af9"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.102931 14102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:59:24.105404 14117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.106642 14102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:59:24.106782 14102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "0938835f982e4bff9bd647ed55764c80"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "99c67bc080307abb5134b0bfcecd358c"
server_key_iv: "52fbd9388d395d020f9f2645f3560af9"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.106894 14102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:24.127547 14102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:24.130939 14102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:24.131166 14102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:24.138299 14102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40705
I20260929 00:59:24.138592 14169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40705 every 8 connection(s)
I20260929 00:59:24.139333 14102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:24.142031 14170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:24.147071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 14102
I20260929 00:59:24.147207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:59:24.147433 14170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Bootstrap starting.
I20260929 00:59:24.147511 392 external_mini_cluster.cc:1428] Setting key b3ec51eaaa1a50917b1e9a95e4e71fa6
I20260929 00:59:24.149899 14170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:24.150679 14170 log.cc:824] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:24.152467 14170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: No bootstrap required, opened a new log
Sep 29 00:59:24 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:24.155917 14170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:24.156188 14170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:24.156248 14170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0938835f982e4bff9bd647ed55764c80, State: Initialized, Role: FOLLOWER
I20260929 00:59:24.156739 14170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:24.156874 14170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:24.156968 14170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:24.157119 14170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:24.157974 14170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:24.158345 14170 leader_election.cc:302] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0938835f982e4bff9bd647ed55764c80; no voters:
I20260929 00:59:24.158648 14170 leader_election.cc:288] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:24.158754 14175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:24.158954 14175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 1 LEADER]: Becoming Leader. State: Replica: 0938835f982e4bff9bd647ed55764c80, State: Running, Role: LEADER
I20260929 00:59:24.159335 14175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:24.160019 14170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:24.161087 14176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0938835f982e4bff9bd647ed55764c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } } }
I20260929 00:59:24.161365 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0938835f982e4bff9bd647ed55764c80. Latest consensus state: current_term: 1 leader_uuid: "0938835f982e4bff9bd647ed55764c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } } }
I20260929 00:59:24.161473 14176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:24.161759 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:24.162119 14186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:24.162276 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.148864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.0.98.62:40705)
0929 00:59:24.149268 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:24.149283 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:24.149318 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:59:24.149899 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:24.149915 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:24.150362 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:24.150702 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:24.151538 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.152492 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.153375 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.153849 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.154023 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.156697 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:24.156702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:24.156714 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:24.156739 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:24.158861 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.159399 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.159405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.159410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.159548 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.159857 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.159860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.159862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.160374 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:24.160571 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:24.160864 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 00:59:24.161183 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260929 00:59:24.165429 14186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:24.170523 14186 catalog_manager.cc:1347] Generated new cluster ID: 2a5c652890ac4eac80bcb8218344b5ea
I20260929 00:59:24.170588 14186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:24.186481 14186 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:24.187947 14186 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:24.192716 14186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Generated new TSK 0
I20260929 00:59:24.193578 14186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:59:24.229545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40705
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:24.354054 14196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:24.354319 14196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:24.354362 14196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:24.358309 14196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:24.358357 14196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:24.358412 14196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:59:24.363682 14196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:24.364589 14196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:24.365603 14196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:24.375303 14202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:24.375393 14196 server_base.cc:1034] running on GCE node
W20260929 00:59:24.375334 14201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:24.375257 14204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:24.375885 14196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:24.376495 14196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:24.377689 14196 hybrid_clock.cc:648] HybridClock initialized: now 1790643564377669 us; error 28 us; skew 500 ppm
Sep 29 00:59:24 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:24.380410 14196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:59:24.381260 14196 webserver.cc:460] Webserver started at http://127.0.98.1:42741/ using document root <none> and password file <none>
I20260929 00:59:24.381822 14196 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:24.381912 14196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:24.382138 14196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:24.384344 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "0e1e8b7eda924d0b86e34102a0f38d37"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "080c9846f1806b09e2ae9b517dd877b7"
server_key_iv: "091af60ae198bf959fcb6b3886d3795a"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.384910 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0e1e8b7eda924d0b86e34102a0f38d37"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "080c9846f1806b09e2ae9b517dd877b7"
server_key_iv: "091af60ae198bf959fcb6b3886d3795a"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.388526 14196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:59:24.390913 14211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.392053 14196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:24.392189 14196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "0e1e8b7eda924d0b86e34102a0f38d37"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "080c9846f1806b09e2ae9b517dd877b7"
server_key_iv: "091af60ae198bf959fcb6b3886d3795a"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.392299 14196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:24.403631 14196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:24.406426 14196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:24.406616 14196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:24.407178 14196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:24.407963 14196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:24.408033 14196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.408102 14196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:24.408156 14196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.416792 14196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34163
I20260929 00:59:24.416880 14324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34163 every 8 connection(s)
I20260929 00:59:24.417686 14196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:24.426506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 14196
I20260929 00:59:24.426674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:59:24.426978 392 external_mini_cluster.cc:1428] Setting key 2226b26cdbaa4123c884b17b57f25d9d
Sep 29 00:59:24 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:24.431443 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.419470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55003 (local address 127.0.98.62:40705)
0929 00:59:24.419632 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:24.419636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:24.420442 (+ 806us) server_negotiation.cc:407] Connection header received
0929 00:59:24.421664 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:24.421667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:24.421720 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:24.421796 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:24.423076 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.423611 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.424732 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.424959 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.425035 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.427870 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:24.427874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:24.427876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:24.427900 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:24.429360 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.429881 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.429886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.429887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.429968 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.430374 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.430379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.430381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.430560 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:24.430668 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:24.431162 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 00:59:24.431285 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:59:24.432256 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.419861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:55003)
0929 00:59:24.420291 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:24.420333 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:24.421473 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:24.421953 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:24.421959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:24.422479 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:24.422945 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.422958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.423718 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.423720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.424616 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.424625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.425084 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.425087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.425237 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.425845 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:24.425861 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:24.427687 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:24.429496 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:24.429502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:24.429524 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:24.429776 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:24.430062 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:24.430066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:24.430068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:24.430279 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:24.430667 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:24.430674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:24.430924 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 00:59:24.431107 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 00:59:24.431438 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260929 00:59:24.433470 14325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40705
I20260929 00:59:24.433748 14325 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:24.434376 14325 heartbeater.cc:511] Master 127.0.98.62:40705 requested a full tablet report, sending...
I20260929 00:59:24.435886 14134 ts_manager.cc:194] Registered new tserver with Master: 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:59:24.437021 14134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:59:24.453622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40705
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:24.562456 14330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:24.562712 14330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:24.562785 14330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:24.566771 14330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:24.566833 14330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:24.566943 14330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:59:24.572240 14330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:24.573267 14330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:24.574285 14330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:24.583650 14338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:24.583669 14335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:24.583707 14336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:24.583855 14330 server_base.cc:1034] running on GCE node
I20260929 00:59:24.584328 14330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:24.584935 14330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:24.586095 14330 hybrid_clock.cc:648] HybridClock initialized: now 1790643564586080 us; error 26 us; skew 500 ppm
Sep 29 00:59:24 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:24.588725 14330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:59:24.589591 14330 webserver.cc:460] Webserver started at http://127.0.98.2:39809/ using document root <none> and password file <none>
I20260929 00:59:24.590087 14330 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:24.590144 14330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:24.590386 14330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:24.592298 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "1fa3c54cca2543849e1fdd9f955e8288"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "8023423c1ad0432107dfed8402677cc9"
server_key_iv: "cda90387398150d22cf82baa326fc625"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.592787 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1fa3c54cca2543849e1fdd9f955e8288"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "8023423c1ad0432107dfed8402677cc9"
server_key_iv: "cda90387398150d22cf82baa326fc625"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.596395 14330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:59:24.598719 14345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.599916 14330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:59:24.600037 14330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "1fa3c54cca2543849e1fdd9f955e8288"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "8023423c1ad0432107dfed8402677cc9"
server_key_iv: "cda90387398150d22cf82baa326fc625"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.600145 14330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:24.624689 14330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:24.627765 14330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:24.627965 14330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:24.628561 14330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:24.629413 14330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:24.629473 14330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.629534 14330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:24.629567 14330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.638458 14330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44709
I20260929 00:59:24.638484 14458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44709 every 8 connection(s)
I20260929 00:59:24.639308 14330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:24.639760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 14330
I20260929 00:59:24.639892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:59:24.640185 392 external_mini_cluster.cc:1428] Setting key aa09681630fa690b2df5c7ae284d56e3
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 29 00:59:24 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:59:24.654531 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.641199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55219 (local address 127.0.98.62:40705)
0929 00:59:24.641341 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:24.641346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:24.642285 (+ 939us) server_negotiation.cc:407] Connection header received
0929 00:59:24.643758 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:24.643761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:24.643805 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:24.643875 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:24.645483 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.646077 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.647986 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.648191 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.648276 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.651122 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:24.651126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:24.651130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:24.651159 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:24.652457 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.652997 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.653000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.653001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.653071 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.653413 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.653416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.653418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.653596 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:24.653703 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:24.654234 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 00:59:24.654359 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:59:24.655325 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.641610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:55219)
0929 00:59:24.642115 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:24.642158 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:24.643522 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:24.644073 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:24.644082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:24.644623 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:24.645326 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.645348 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.646237 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.646240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.647867 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.647881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.648346 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.648350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.648566 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.649209 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:24.649234 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:24.650954 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:24.652573 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:24.652578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:24.652593 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:24.652846 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:24.653163 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:24.653167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:24.653169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:24.653313 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:24.653701 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:24.653708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:24.653958 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 00:59:24.654145 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 00:59:24.654492 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":134,"threads_started":1}
I20260929 00:59:24.656481 14459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40705
I20260929 00:59:24.656735 14459 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:24.657222 14459 heartbeater.cc:511] Master 127.0.98.62:40705 requested a full tablet report, sending...
I20260929 00:59:24.658329 14134 ts_manager.cc:194] Registered new tserver with Master: 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709)
I20260929 00:59:24.659173 14134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:59:24.669193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40705
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:24.778594 14464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:24.778851 14464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:24.778895 14464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:24.783340 14464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:24.783401 14464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:24.783488 14464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:59:24.789521 14464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:24.790892 14464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:24.792387 14464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:24.802506 14469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:24.802498 14472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:24.802498 14470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:24.802843 14464 server_base.cc:1034] running on GCE node
I20260929 00:59:24.803265 14464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:24.803957 14464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:24.805145 14464 hybrid_clock.cc:648] HybridClock initialized: now 1790643564805117 us; error 54 us; skew 500 ppm
Sep 29 00:59:24 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:24.807999 14464 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:59:24.808887 14464 webserver.cc:460] Webserver started at http://127.0.98.3:41049/ using document root <none> and password file <none>
I20260929 00:59:24.809464 14464 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:24.809528 14464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:24.809742 14464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:24.811859 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "56ab389c33a34416a50b126e645d3057"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "5f1f82c17f90713f8c04aeb4c84b67b4"
server_key_iv: "e7e68d61a302582a5c396aea425aaa54"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.812439 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56ab389c33a34416a50b126e645d3057"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "5f1f82c17f90713f8c04aeb4c84b67b4"
server_key_iv: "e7e68d61a302582a5c396aea425aaa54"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.816323 14464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 00:59:24.818765 14479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.820009 14464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:59:24.820142 14464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "56ab389c33a34416a50b126e645d3057"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "5f1f82c17f90713f8c04aeb4c84b67b4"
server_key_iv: "e7e68d61a302582a5c396aea425aaa54"
server_key_version: "encryptionkey@0"
I20260929 00:59:24.820254 14464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:24.845471 14464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:24.848502 14464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:24.848706 14464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:24.849442 14464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:24.850456 14464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:24.850502 14464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.850569 14464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:24.850605 14464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:24.865571 14464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42165
I20260929 00:59:24.865653 14592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42165 every 8 connection(s)
I20260929 00:59:24.866969 14464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:24.876035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 14464
I20260929 00:59:24.876174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:59:24.876430 392 external_mini_cluster.cc:1428] Setting key 7535a8eb55ba5b15a62e849ee2614d9e
Sep 29 00:59:24 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:24.884923 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.869025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46487 (local address 127.0.98.62:40705)
0929 00:59:24.869159 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:24.869163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:24.870266 (+ 1103us) server_negotiation.cc:407] Connection header received
0929 00:59:24.871999 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:24.872002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:24.872063 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:24.872145 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:24.873314 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.873983 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.875938 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.876236 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.876337 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.880117 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:24.880123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:24.880130 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:24.880168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:24.882053 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.882792 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.882795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.882797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.882898 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.883364 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.883368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.883369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.883659 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:24.883763 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:24.884529 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 00:59:24.884708 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:59:24.886117 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.869551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:46487)
0929 00:59:24.870072 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:24.870138 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:24.871760 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:24.872327 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:24.872334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:24.872690 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:24.873163 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.873177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.874109 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.874113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.875801 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.875827 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.876376 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.876380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.876606 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.877804 (+ 1198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:24.877825 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:24.879910 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:24.882225 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:24.882234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:24.882251 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:24.882660 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:24.883027 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:24.883032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:24.883035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:24.883240 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:24.883780 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:24.883790 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:24.884182 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 00:59:24.884438 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 00:59:24.884880 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260929 00:59:24.887596 14593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40705
I20260929 00:59:24.887879 14593 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:24.888429 14593 heartbeater.cc:511] Master 127.0.98.62:40705 requested a full tablet report, sending...
I20260929 00:59:24.889667 14134 ts_manager.cc:194] Registered new tserver with Master: 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165)
I20260929 00:59:24.890508 14134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46487
I20260929 00:59:24.890921 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:24.901821 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.892921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.0.98.62:40705)
0929 00:59:24.893041 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:24.893046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:24.893363 (+ 317us) server_negotiation.cc:407] Connection header received
0929 00:59:24.893503 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:24.893509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:24.893580 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:24.893670 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:24.894620 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.895542 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.896422 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.896856 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.896983 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.898518 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:24.898522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:24.898525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:24.898558 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:24.900059 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.900511 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.900515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.900517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.900599 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:24.900898 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:24.900901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:24.900903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:24.901161 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:24.901299 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:24.901544 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 00:59:24.901660 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:59:24.904949 14134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:59:24.908522 14134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:59:24.923545 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.917985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.0.98.2:44709)
0929 00:59:24.918455 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:24.918466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:24.918490 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:59:24.918585 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:24.918591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:24.918828 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:24.918997 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:24.919401 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.920157 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.921877 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.922684 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.922817 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.922980 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:24.923168 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 00:59:24.923383 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":114,"threads_started":1}
I20260929 00:59:24.923539 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.917196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44709 (local address 127.0.0.1:32770)
0929 00:59:24.917971 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:24.917985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:24.918082 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:24.919164 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:24.919166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:24.919189 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:24.919276 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.919280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.920294 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.920299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.921672 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.921683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.922819 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.922822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.922934 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.922972 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 00:59:24.923120 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 00:59:24.923384 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":3,"spinlock_wait_cycles":1408,"thread_start_us":110,"threads_started":1}
I20260929 00:59:24.924089 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.917166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34163 (local address 127.0.0.1:44084)
0929 00:59:24.917787 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:24.917840 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:24.917972 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:24.919019 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:24.919025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:24.919051 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:24.919185 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.919192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.920888 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.920892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.922224 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.922235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.923362 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.923368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.923539 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.923561 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:59:24.923622 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 00:59:24.923954 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":116,"thread_start_us":76,"threads_started":1}
I20260929 00:59:24.924551 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.917303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.0.98.1:34163)
0929 00:59:24.918579 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:24.918594 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:24.918612 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:59:24.918686 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:24.918690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:24.918846 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:24.919351 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:24.919531 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.920673 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.922350 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.923148 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.923568 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.924188 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:24.924249 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 00:59:24.924363 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":82,"threads_started":1}
I20260929 00:59:24.924750 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.917511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42165 (local address 127.0.0.1:49466)
0929 00:59:24.918347 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:24.918361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:24.918460 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:24.919167 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:24.919169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:24.919196 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:24.919299 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.919306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.920783 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.920787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.922039 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:24.922046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.923470 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.923473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:24.924430 (+ 957us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.924459 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:59:24.924547 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:59:24.924628 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":125,"spinlock_wait_cycles":1792,"thread_start_us":85,"threads_started":1}
I20260929 00:59:24.925225 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:24.917944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.0.98.3:42165)
0929 00:59:24.918363 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:24.918377 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:24.918549 (+ 172us) server_negotiation.cc:407] Connection header received
0929 00:59:24.918636 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:24.918647 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:24.918848 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:24.918998 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:24.919437 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.920624 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.922154 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:24.923155 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:24.923267 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:24.923414 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:24.924983 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0929 00:59:24.925084 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":100,"threads_started":1}
I20260929 00:59:24.927107 14393 tablet_service.cc:1467] Processing CreateTablet for tablet 032c83c9bb8a4e8a9324018ac05eeb6a (DEFAULT_TABLE table=test-table [id=a752f72af5e04068954f26a1a6a01760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:24.928117 14393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032c83c9bb8a4e8a9324018ac05eeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:24.927881 14516 tablet_service.cc:1467] Processing CreateTablet for tablet 032c83c9bb8a4e8a9324018ac05eeb6a (DEFAULT_TABLE table=test-table [id=a752f72af5e04068954f26a1a6a01760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:24.928973 14516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032c83c9bb8a4e8a9324018ac05eeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:24.928673 14259 tablet_service.cc:1467] Processing CreateTablet for tablet 032c83c9bb8a4e8a9324018ac05eeb6a (DEFAULT_TABLE table=test-table [id=a752f72af5e04068954f26a1a6a01760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:24.929653 14259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032c83c9bb8a4e8a9324018ac05eeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:24.934420 14610 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: Bootstrap starting.
I20260929 00:59:24.936825 14611 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: Bootstrap starting.
I20260929 00:59:24.936983 14610 tablet_bootstrap.cc:654] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:24.937403 14612 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Bootstrap starting.
I20260929 00:59:24.937824 14610 log.cc:824] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:24.938984 14611 tablet_bootstrap.cc:654] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:24.939466 14612 tablet_bootstrap.cc:654] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:24.939826 14610 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: No bootstrap required, opened a new log
I20260929 00:59:24.940052 14610 ts_tablet_manager.cc:1397] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:59:24.940258 14612 log.cc:824] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:24.940258 14611 log.cc:824] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:24.942308 14612 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: No bootstrap required, opened a new log
I20260929 00:59:24.942499 14612 ts_tablet_manager.cc:1397] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:59:24.942560 14611 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: No bootstrap required, opened a new log
I20260929 00:59:24.942725 14611 ts_tablet_manager.cc:1397] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:59:24.943535 14610 raft_consensus.cc:348] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:24.943794 14610 raft_consensus.cc:374] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:24.943845 14610 raft_consensus.cc:729] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa3c54cca2543849e1fdd9f955e8288, State: Initialized, Role: FOLLOWER
I20260929 00:59:24.944322 14610 consensus_queue.cc:260] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:24.945185 14610 ts_tablet_manager.cc:1428] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260929 00:59:24.945554 14459 heartbeater.cc:503] Master 127.0.98.62:40705 was elected leader, sending a full tablet report...
I20260929 00:59:24.946509 14611 raft_consensus.cc:348] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:24.946794 14611 raft_consensus.cc:374] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:24.946848 14611 raft_consensus.cc:729] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ab389c33a34416a50b126e645d3057, State: Initialized, Role: FOLLOWER
I20260929 00:59:24.946717 14612 raft_consensus.cc:348] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:24.946969 14612 raft_consensus.cc:374] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:24.947018 14612 raft_consensus.cc:729] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1e8b7eda924d0b86e34102a0f38d37, State: Initialized, Role: FOLLOWER
I20260929 00:59:24.947314 14611 consensus_queue.cc:260] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:24.947371 14612 consensus_queue.cc:260] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:24.947966 14325 heartbeater.cc:503] Master 127.0.98.62:40705 was elected leader, sending a full tablet report...
I20260929 00:59:24.948252 14593 heartbeater.cc:503] Master 127.0.98.62:40705 was elected leader, sending a full tablet report...
I20260929 00:59:24.948266 14611 ts_tablet_manager.cc:1428] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:59:24.949188 14612 ts_tablet_manager.cc:1428] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 00:59:25.020785 14616 raft_consensus.cc:482] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:25.021020 14616 raft_consensus.cc:504] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:25.022250 14616 leader_election.cc:288] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163), 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165)
I20260929 00:59:25.026449 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.022632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41279 (local address 127.0.98.3:42165)
0929 00:59:25.022761 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.022765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.022781 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:59:25.022853 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.022856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.022906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.022971 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:25.023548 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.024152 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.025374 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.025881 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.025955 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.026008 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.026255 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.026310 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:59:25.026470 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.022530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42165 (local address 127.0.98.2:41279)
0929 00:59:25.022650 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.022668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:25.022759 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:25.023050 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:25.023056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:25.023134 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:25.023280 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.023286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.024267 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.024270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.025252 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.025261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.025990 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.025994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.026098 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.026123 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:59:25.026237 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 00:59:25.026314 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 00:59:25.026909 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.022720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49915 (local address 127.0.98.1:34163)
0929 00:59:25.022853 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.022858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.023080 (+ 222us) server_negotiation.cc:407] Connection header received
0929 00:59:25.023272 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.023275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.023318 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.023399 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:25.023684 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.024533 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.025713 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.026261 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.026399 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.026455 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.026686 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.026790 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:59:25.027252 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.022537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34163 (local address 127.0.98.2:49915)
0929 00:59:25.022978 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.023012 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:25.023153 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:25.023409 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:25.023412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:25.023426 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:25.023558 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.023565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.024648 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.024651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.025591 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.025601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.026402 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.026406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.026525 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.026554 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:59:25.027075 (+ 521us) client_negotiation.cc:240] Negotiation successful
0929 00:59:25.027150 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":113,"thread_start_us":135,"threads_started":1}
I20260929 00:59:25.027215 14545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032c83c9bb8a4e8a9324018ac05eeb6a" candidate_uuid: "1fa3c54cca2543849e1fdd9f955e8288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ab389c33a34416a50b126e645d3057" is_pre_election: true
I20260929 00:59:25.027568 14545 raft_consensus.cc:2454] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa3c54cca2543849e1fdd9f955e8288 in term 0.
I20260929 00:59:25.027871 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032c83c9bb8a4e8a9324018ac05eeb6a" candidate_uuid: "1fa3c54cca2543849e1fdd9f955e8288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" is_pre_election: true
I20260929 00:59:25.028092 14348 leader_election.cc:302] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa3c54cca2543849e1fdd9f955e8288, 56ab389c33a34416a50b126e645d3057; no voters:
I20260929 00:59:25.028213 14279 raft_consensus.cc:2454] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa3c54cca2543849e1fdd9f955e8288 in term 0.
I20260929 00:59:25.028393 14616 raft_consensus.cc:2781] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:25.028481 14616 raft_consensus.cc:482] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:25.028549 14616 raft_consensus.cc:3037] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:25.029630 14616 raft_consensus.cc:504] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:25.029974 14616 leader_election.cc:288] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [CANDIDATE]: Term 1 election: Requested vote from peers 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163), 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165)
I20260929 00:59:25.030311 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032c83c9bb8a4e8a9324018ac05eeb6a" candidate_uuid: "1fa3c54cca2543849e1fdd9f955e8288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1e8b7eda924d0b86e34102a0f38d37"
I20260929 00:59:25.030439 14279 raft_consensus.cc:3037] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:25.030462 14545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032c83c9bb8a4e8a9324018ac05eeb6a" candidate_uuid: "1fa3c54cca2543849e1fdd9f955e8288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ab389c33a34416a50b126e645d3057"
I20260929 00:59:25.030612 14545 raft_consensus.cc:3037] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:25.031517 14279 raft_consensus.cc:2454] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa3c54cca2543849e1fdd9f955e8288 in term 1.
I20260929 00:59:25.031672 14545 raft_consensus.cc:2454] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa3c54cca2543849e1fdd9f955e8288 in term 1.
I20260929 00:59:25.031796 14349 leader_election.cc:302] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1e8b7eda924d0b86e34102a0f38d37, 1fa3c54cca2543849e1fdd9f955e8288; no voters:
I20260929 00:59:25.031972 14616 raft_consensus.cc:2781] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:25.032230 14616 raft_consensus.cc:686] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa3c54cca2543849e1fdd9f955e8288, State: Running, Role: LEADER
I20260929 00:59:25.032594 14616 consensus_queue.cc:237] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:25.036114 14133 catalog_manager.cc:5495] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1fa3c54cca2543849e1fdd9f955e8288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } health_report { overall_health: UNKNOWN } } }
I20260929 00:59:25.050175 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.046627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.0.98.2:44709)
0929 00:59:25.046765 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.046770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.046785 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:59:25.046904 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.046907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.046963 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.047080 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:25.047333 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.047864 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.048722 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.048938 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.049048 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.049359 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:25.049777 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:25.049915 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.049957 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.050012 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:59:25.055961 14545 raft_consensus.cc:1261] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 1 FOLLOWER]: Refusing update from remote peer 1fa3c54cca2543849e1fdd9f955e8288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:25.055984 14279 raft_consensus.cc:1261] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Refusing update from remote peer 1fa3c54cca2543849e1fdd9f955e8288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:25.056654 14616 consensus_queue.cc:1035] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:25.056818 14620 consensus_queue.cc:1035] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:25.065059 14625 mvcc.cc:205] Tried to move back new op lower bound from 7334476042461589504 to 7334476042375172096. Current Snapshot: MvccSnapshot[applied={T|T < 7334476042461589504}]
I20260929 00:59:25.065224 14623 mvcc.cc:205] Tried to move back new op lower bound from 7334476042461589504 to 7334476042375172096. Current Snapshot: MvccSnapshot[applied={T|T < 7334476042461589504}]
I20260929 00:59:25.067338 14624 mvcc.cc:205] Tried to move back new op lower bound from 7334476042461589504 to 7334476042375172096. Current Snapshot: MvccSnapshot[applied={T|T < 7334476042461589504}]
I20260929 00:59:25.072361 14133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:25.072496 14133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:25.074887 14133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Sending DeleteTablet for 3 replicas of tablet 032c83c9bb8a4e8a9324018ac05eeb6a
I20260929 00:59:25.075613 14516 tablet_service.cc:1514] Processing DeleteTablet for tablet 032c83c9bb8a4e8a9324018ac05eeb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49466
I20260929 00:59:25.075613 14393 tablet_service.cc:1514] Processing DeleteTablet for tablet 032c83c9bb8a4e8a9324018ac05eeb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32770
I20260929 00:59:25.076084 14634 tablet_replica.cc:330] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: stopping tablet replica
I20260929 00:59:25.076094 14259 tablet_service.cc:1514] Processing DeleteTablet for tablet 032c83c9bb8a4e8a9324018ac05eeb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44084
I20260929 00:59:25.076395 14633 tablet_replica.cc:330] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: stopping tablet replica
I20260929 00:59:25.076400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 14196
I20260929 00:59:25.076696 14633 raft_consensus.cc:2229] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:25.076995 14633 raft_consensus.cc:2258] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:25.077425 14634 raft_consensus.cc:2229] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:59:25.077744 14634 raft_consensus.cc:2258] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:25.079279 14634 ts_tablet_manager.cc:1905] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:25.079272 14633 ts_tablet_manager.cc:1905] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:25.082345 14633 ts_tablet_manager.cc:1918] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:25.082458 14633 log.cc:1196] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/032c83c9bb8a4e8a9324018ac05eeb6a
I20260929 00:59:25.082832 14633 ts_tablet_manager.cc:1939] T 032c83c9bb8a4e8a9324018ac05eeb6a P 56ab389c33a34416a50b126e645d3057: Deleting consensus metadata
I20260929 00:59:25.082993 14634 ts_tablet_manager.cc:1918] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:25.083140 14634 log.cc:1196] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/032c83c9bb8a4e8a9324018ac05eeb6a
I20260929 00:59:25.083539 14634 ts_tablet_manager.cc:1939] T 032c83c9bb8a4e8a9324018ac05eeb6a P 1fa3c54cca2543849e1fdd9f955e8288: Deleting consensus metadata
I20260929 00:59:25.084182 14120 catalog_manager.cc:4841] TS 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165): tablet 032c83c9bb8a4e8a9324018ac05eeb6a (table test-table [id=a752f72af5e04068954f26a1a6a01760]) successfully deleted
I20260929 00:59:25.084721 14118 catalog_manager.cc:4841] TS 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709): tablet 032c83c9bb8a4e8a9324018ac05eeb6a (table test-table [id=a752f72af5e04068954f26a1a6a01760]) successfully deleted
W20260929 00:59:25.084717 14121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34163 (error 108)
I20260929 00:59:25.085157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 14330
I20260929 00:59:25.087347 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.087070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34163 (local address 127.0.0.1:44086)
0929 00:59:25.087208 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.087275 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:59:25.087540 14121 catalog_manager.cc:4584] TS 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 032c83c9bb8a4e8a9324018ac05eeb6a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34163: connect: Connection refused (error 111)
I20260929 00:59:25.090708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 14464
I20260929 00:59:25.096113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 14102
I20260929 00:59:25.102820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40705
--webserver_interface=127.0.98.62
--webserver_port=46045
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:25.211439 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:25.211726 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:25.211769 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:25.215898 14637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:25.215963 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:25.215983 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:25.216001 14637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:25.216015 14637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:25.221313 14637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:25.222462 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:25.223500 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:25.236550 14644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.236550 14643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.236575 14646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:25.236860 14637 server_base.cc:1034] running on GCE node
I20260929 00:59:25.237399 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:25.238680 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:25.239842 14637 hybrid_clock.cc:648] HybridClock initialized: now 1790643565239825 us; error 30 us; skew 500 ppm
Sep 29 00:59:25 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:25.248345 14637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:59:25.249367 14637 webserver.cc:460] Webserver started at http://127.0.98.62:46045/ using document root <none> and password file <none>
I20260929 00:59:25.249985 14637 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:25.250061 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:25.253997 14637 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:59:25.256202 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.257468 14637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:25.257628 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "0938835f982e4bff9bd647ed55764c80"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "99c67bc080307abb5134b0bfcecd358c"
server_key_iv: "52fbd9388d395d020f9f2645f3560af9"
server_key_version: "encryptionkey@0"
I20260929 00:59:25.258298 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:25.277508 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:25.280599 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:25.280772 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:25.287920 14637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40705
I20260929 00:59:25.288209 14705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40705 every 8 connection(s)
I20260929 00:59:25.289136 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:25.293715 14706 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:59:25.295537 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Bootstrap starting.
I20260929 00:59:25.299343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 14637
I20260929 00:59:25.299798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34163
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40705
--webserver_port=42741
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:25.317180 14706 log.cc:824] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:25.328682 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:59:25.329154 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Bootstrap complete.
I20260929 00:59:25.334625 14706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:25.335150 14706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0938835f982e4bff9bd647ed55764c80, State: Initialized, Role: FOLLOWER
I20260929 00:59:25.335729 14706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:25.335886 14706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:25.335958 14706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:25.336054 14706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:59:25.338591 14706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:25.338973 14706 leader_election.cc:302] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0938835f982e4bff9bd647ed55764c80; no voters:
I20260929 00:59:25.339326 14706 leader_election.cc:288] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:59:25.339475 14710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:59:25.339746 14710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 2 LEADER]: Becoming Leader. State: Replica: 0938835f982e4bff9bd647ed55764c80, State: Running, Role: LEADER
I20260929 00:59:25.340070 14710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:25.340634 14706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:25.341095 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0938835f982e4bff9bd647ed55764c80. Latest consensus state: current_term: 2 leader_uuid: "0938835f982e4bff9bd647ed55764c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } } }
I20260929 00:59:25.341250 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:25.341483 14711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0938835f982e4bff9bd647ed55764c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } } }
I20260929 00:59:25.341562 14711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:25.341678 14715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:25.346561 14715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:25.347383 14715 catalog_manager.cc:1259] Loaded cluster ID: 2a5c652890ac4eac80bcb8218344b5ea
I20260929 00:59:25.347460 14715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:25.348554 14715 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 00:59:25.351459 14715 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 00:59:25.355829 14715 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:25.356541 14715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Loaded TSK: 0
I20260929 00:59:25.357523 14715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:59:25.436106 14708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:25.436359 14708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:25.436431 14708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:25.440527 14708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:25.440601 14708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:25.440697 14708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:59:25.446465 14708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42741
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:25.447455 14708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:25.448457 14708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:25.459199 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.459185 14733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.459455 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:25.459511 14708 server_base.cc:1034] running on GCE node
I20260929 00:59:25.460090 14708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:25.460896 14708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:25.462101 14708 hybrid_clock.cc:648] HybridClock initialized: now 1790643565462084 us; error 39 us; skew 500 ppm
Sep 29 00:59:25 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:25.465413 14708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:59:25.466408 14708 webserver.cc:460] Webserver started at http://127.0.98.1:42741/ using document root <none> and password file <none>
I20260929 00:59:25.466970 14708 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:25.467049 14708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:25.471062 14708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:59:25.473686 14743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.475216 14708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:59:25.475339 14708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "0e1e8b7eda924d0b86e34102a0f38d37"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "080c9846f1806b09e2ae9b517dd877b7"
server_key_iv: "091af60ae198bf959fcb6b3886d3795a"
server_key_version: "encryptionkey@0"
I20260929 00:59:25.476053 14708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:25.527057 14708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:25.530246 14708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:25.530409 14708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:25.530973 14708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:25.532023 14750 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 00:59:25.536175 14708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 00:59:25.536242 14708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 00:59:25.536325 14708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 00:59:25.538030 14708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 00:59:25.538074 14708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 00:59:25.538486 14750 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Bootstrap starting.
I20260929 00:59:25.548590 14708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34163
I20260929 00:59:25.548694 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34163 every 8 connection(s)
I20260929 00:59:25.549523 14708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:25.557874 14750 log.cc:824] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:25.558317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 14708
I20260929 00:59:25.558794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44709
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40705
--webserver_port=39809
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:59:25 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:25.567265 14750 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:59:25.567353 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.551519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43957 (local address 127.0.98.62:40705)
0929 00:59:25.552034 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.552051 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.552100 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:59:25.553222 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.553237 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.553641 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.554060 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:25.555089 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.556792 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.557951 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.558650 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.558845 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.561828 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:25.561833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:25.561845 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:25.561872 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:25.564129 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.564703 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.564708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.564714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.564839 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.565191 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.565194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.565196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.565536 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:25.565684 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.566257 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.566540 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":124,"threads_started":1}
I20260929 00:59:25.567364 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.551463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:43957)
0929 00:59:25.551932 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.551985 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:25.552779 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:25.554057 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:25.554064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:25.554470 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:25.554918 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.554931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.556952 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.556954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.557824 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.557832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.558801 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.558806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.559000 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.559699 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:25.559721 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:25.561637 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:25.564305 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:25.564311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:25.564324 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:25.564581 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:25.564941 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:25.564946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:25.564948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:25.565086 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:25.565656 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:25.565662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:25.565928 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 00:59:25.566113 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 00:59:25.566467 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":146,"threads_started":1}
I20260929 00:59:25.567780 14750 tablet_bootstrap.cc:492] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Bootstrap complete.
I20260929 00:59:25.569324 14750 ts_tablet_manager.cc:1397] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.008s
I20260929 00:59:25.569581 14858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40705
I20260929 00:59:25.569829 14858 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:25.570364 14858 heartbeater.cc:511] Master 127.0.98.62:40705 requested a full tablet report, sending...
I20260929 00:59:25.572594 14670 ts_manager.cc:194] Registered new tserver with Master: 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163)
I20260929 00:59:25.573002 14750 raft_consensus.cc:348] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:25.573518 14750 raft_consensus.cc:729] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1e8b7eda924d0b86e34102a0f38d37, State: Initialized, Role: FOLLOWER
I20260929 00:59:25.574000 14750 consensus_queue.cc:260] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } } peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } }
I20260929 00:59:25.575006 14750 ts_tablet_manager.cc:1428] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20260929 00:59:25.576615 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43957
I20260929 00:59:25.579192 14858 heartbeater.cc:503] Master 127.0.98.62:40705 was elected leader, sending a full tablet report...
Sep 29 00:59:25 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 00:59:25.587618 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.576066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34163 (local address 127.0.0.1:44102)
0929 00:59:25.577059 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.577105 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:25.577242 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:25.577782 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:25.577789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:25.578297 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:25.578442 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.578452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.579934 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.579938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.580935 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.580947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.581758 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.581774 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.581856 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.582520 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:25.582532 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:25.583933 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:25.585966 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:25.585973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:25.585978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:25.586238 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:25.586554 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:25.586556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:25.586557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:25.586631 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:25.587129 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:25.587134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:25.587231 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 00:59:25.587343 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 00:59:25.587487 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":108,"threads_started":1}
I20260929 00:59:25.587793 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.576175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.0.98.1:34163)
0929 00:59:25.576820 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.576831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.577217 (+ 386us) server_negotiation.cc:407] Connection header received
0929 00:59:25.577416 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.577421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.577622 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.577756 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:25.578586 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.579836 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.581060 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.581653 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.581756 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.584067 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:25.584076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:25.584089 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:25.584118 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:25.585857 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.586360 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.586366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.586368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.586462 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.586724 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.586728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.586729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.587018 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:25.587152 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.587462 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.587634 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":546,"thread_start_us":103,"threads_started":1}
I20260929 00:59:25.588780 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet 032c83c9bb8a4e8a9324018ac05eeb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44102
I20260929 00:59:25.589600 14867 tablet_replica.cc:330] stopping tablet replica
I20260929 00:59:25.589898 14867 raft_consensus.cc:2229] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:25.590143 14867 raft_consensus.cc:2258] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:25.591163 14867 ts_tablet_manager.cc:1905] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:25.594789 14867 ts_tablet_manager.cc:1918] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:25.594901 14867 log.cc:1196] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/032c83c9bb8a4e8a9324018ac05eeb6a
I20260929 00:59:25.595172 14867 ts_tablet_manager.cc:1939] T 032c83c9bb8a4e8a9324018ac05eeb6a P 0e1e8b7eda924d0b86e34102a0f38d37: Deleting consensus metadata
I20260929 00:59:25.596856 14657 catalog_manager.cc:4841] TS 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163): tablet 032c83c9bb8a4e8a9324018ac05eeb6a (table test-table [id=a752f72af5e04068954f26a1a6a01760]) successfully deleted
W20260929 00:59:25.679579 14863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:25.679838 14863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:25.679881 14863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:25.683979 14863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:25.684044 14863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:25.684103 14863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:59:25.689306 14863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39809
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:25.690397 14863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:25.691422 14863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:59:25.701294 14863 server_base.cc:1034] running on GCE node
W20260929 00:59:25.701232 14873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.701232 14872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.701226 14875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:25.701915 14863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:25.702621 14863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:25.703788 14863 hybrid_clock.cc:648] HybridClock initialized: now 1790643565703755 us; error 43 us; skew 500 ppm
Sep 29 00:59:25 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:25.706497 14863 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:59:25.707376 14863 webserver.cc:460] Webserver started at http://127.0.98.2:39809/ using document root <none> and password file <none>
I20260929 00:59:25.707917 14863 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:25.708011 14863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:25.711735 14863 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:59:25.713718 14882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.714950 14863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:25.715081 14863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "1fa3c54cca2543849e1fdd9f955e8288"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "8023423c1ad0432107dfed8402677cc9"
server_key_iv: "cda90387398150d22cf82baa326fc625"
server_key_version: "encryptionkey@0"
I20260929 00:59:25.715662 14863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:25.744793 14863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:25.748270 14863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:25.748498 14863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:25.749099 14863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:25.749903 14863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:25.749949 14863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.750017 14863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:25.750053 14863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.758625 14863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44709
I20260929 00:59:25.758711 14995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44709 every 8 connection(s)
I20260929 00:59:25.759658 14863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:25.766441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 14863
I20260929 00:59:25.766938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42165
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40705
--webserver_port=41049
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:59:25 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:25.774271 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.761509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34941 (local address 127.0.98.62:40705)
0929 00:59:25.761667 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.761671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.762616 (+ 945us) server_negotiation.cc:407] Connection header received
0929 00:59:25.763961 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.763964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.764033 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.764111 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:25.765307 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.765845 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.767604 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.767849 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.767947 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.770635 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:25.770638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:25.770643 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:25.770677 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:25.772153 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.772690 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.772694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.772695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.772778 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.773129 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.773132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.773134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.773326 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:25.773433 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.773930 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.774066 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:59:25.775044 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.761929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:34941)
0929 00:59:25.762462 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.762506 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:25.763719 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:25.764285 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:25.764292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:25.764662 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:25.765140 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.765153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.765977 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.765981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.767461 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.767476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.768027 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.768033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.768278 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.768959 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:25.768975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:25.770463 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:25.772291 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:25.772297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:25.772312 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:25.772578 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:25.772880 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:25.772883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:25.772885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:25.773026 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:25.773433 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:25.773441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:25.773682 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 00:59:25.773872 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 00:59:25.774218 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":114,"threads_started":1}
I20260929 00:59:25.776247 14996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40705
I20260929 00:59:25.776507 14996 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:25.777000 14996 heartbeater.cc:511] Master 127.0.98.62:40705 requested a full tablet report, sending...
I20260929 00:59:25.778033 14670 ts_manager.cc:194] Registered new tserver with Master: 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709)
I20260929 00:59:25.778798 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34941
W20260929 00:59:25.877565 14999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:25.877812 14999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:25.877854 14999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:25.882045 14999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:25.882105 14999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:25.882164 14999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:59:25.887444 14999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41049
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:25.888422 14999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:25.889395 14999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:25.899300 15005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.899328 15004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:25.899360 14999 server_base.cc:1034] running on GCE node
W20260929 00:59:25.899304 15007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:25.900110 14999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:25.900688 14999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:25.901889 14999 hybrid_clock.cc:648] HybridClock initialized: now 1790643565901874 us; error 33 us; skew 500 ppm
Sep 29 00:59:25 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:25.904582 14999 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:59:25.905478 14999 webserver.cc:460] Webserver started at http://127.0.98.3:41049/ using document root <none> and password file <none>
I20260929 00:59:25.906036 14999 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:25.906147 14999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:25.909807 14999 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:59:25.911832 15014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.912979 14999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:59:25.913111 14999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "56ab389c33a34416a50b126e645d3057"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "5f1f82c17f90713f8c04aeb4c84b67b4"
server_key_iv: "e7e68d61a302582a5c396aea425aaa54"
server_key_version: "encryptionkey@0"
I20260929 00:59:25.913643 14999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:25.936620 14999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:25.939613 14999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:25.939826 14999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:25.940426 14999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:25.941288 14999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:25.941358 14999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.941427 14999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:25.941496 14999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.950093 14999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42165
I20260929 00:59:25.950127 15127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42165 every 8 connection(s)
I20260929 00:59:25.951097 14999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:25.953471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 14999
Sep 29 00:59:25 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:25.963436 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.954195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.0.98.62:40705)
0929 00:59:25.954457 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.954462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.954567 (+ 105us) server_negotiation.cc:407] Connection header received
0929 00:59:25.954727 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.954730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.954787 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.954887 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:25.955693 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.956229 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.957390 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.957747 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.957858 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.959497 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:25.959500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:25.959503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:25.959530 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:25.961244 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.961739 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.961743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.961745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.961833 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.962115 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.962119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.962121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.962430 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:25.962693 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.963145 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.963253 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260929 00:59:25.965817 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.952863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46135 (local address 127.0.98.62:40705)
0929 00:59:25.953044 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.953048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.954011 (+ 963us) server_negotiation.cc:407] Connection header received
0929 00:59:25.955382 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.955385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.955448 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.955538 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:25.956649 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.957367 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.958764 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.959155 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.959279 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.962004 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:25.962008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:25.962010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:25.962034 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:25.963686 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.964279 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.964284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.964285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.964367 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.964723 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.964728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.964729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.964891 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:25.964993 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.965575 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.965681 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 00:59:25.966759 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.953269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:46135)
0929 00:59:25.953797 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.953866 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:25.955179 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:25.955699 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:25.955706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:25.956054 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:25.956519 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.956531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.957512 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.957515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.958640 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.958650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.959303 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.959306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.959498 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.960077 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:25.960099 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:25.961826 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:25.963820 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:25.963826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:25.963839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:25.964187 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:25.964474 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:25.964477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:25.964479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:25.964624 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:25.965007 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:25.965015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:25.965340 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 00:59:25.965534 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 00:59:25.965880 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":130,"threads_started":1}
I20260929 00:59:25.967803 15128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40705
I20260929 00:59:25.968070 15128 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:25.968590 15128 heartbeater.cc:511] Master 127.0.98.62:40705 requested a full tablet report, sending...
I20260929 00:59:25.969588 14670 ts_manager.cc:194] Registered new tserver with Master: 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165)
I20260929 00:59:25.970141 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46135
I20260929 00:59:25.976778 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:25.980794 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.977359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.0.98.62:40705)
0929 00:59:25.977465 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.977469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.977715 (+ 246us) server_negotiation.cc:407] Connection header received
0929 00:59:25.977830 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.977832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.977921 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.978005 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:25.978267 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.978764 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.979568 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.979763 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.979834 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.980095 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:25.980467 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:25.980593 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.980629 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.980671 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 00:59:25.981392 14670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:59:25.984700 14670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:59:25.992952 14792 tablet_service.cc:1467] Processing CreateTablet for tablet 630bd374aa0d41218f277293a1efcc14 (DEFAULT_TABLE table=test-table [id=0fca1e3f14fe44fbb1f8b0e294633f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:25.993670 14792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630bd374aa0d41218f277293a1efcc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:25.996295 14750 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Bootstrap starting.
I20260929 00:59:25.997236 14750 tablet_bootstrap.cc:654] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:25.998071 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.991626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44709 (local address 127.0.0.1:32790)
0929 00:59:25.991764 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.991780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:25.991866 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:25.992895 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:25.992898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:25.992917 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:25.993026 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.993032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.994841 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.994847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.996402 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.996409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.997504 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.997508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.997692 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.997750 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 00:59:25.997837 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 00:59:25.997934 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":236}
I20260929 00:59:25.998073 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.991836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.0.98.2:44709)
0929 00:59:25.992221 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.992240 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.992266 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:59:25.992414 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.992422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.992733 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.992925 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:25.993132 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.994697 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.996595 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.997338 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.997480 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.997634 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.997881 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.997961 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":135,"threads_started":1}
I20260929 00:59:25.998799 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.991886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42165 (local address 127.0.0.1:49474)
0929 00:59:25.992313 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:25.992328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:25.992419 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:25.993321 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:25.993325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:25.993345 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:25.993441 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.993447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.994980 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.994983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.996260 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:25.996270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.997424 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.997431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:25.997542 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.998528 (+ 986us) client_negotiation.cc:769] Sending connection context
0929 00:59:25.998608 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 00:59:25.998675 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1009,"thread_start_us":103,"threads_started":1}
I20260929 00:59:25.998904 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.992377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.0.98.3:42165)
0929 00:59:25.992812 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.992826 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.992859 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:59:25.992932 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.992937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.993131 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.993593 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:25.993749 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.994846 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.996409 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.997261 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.997588 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.997719 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.998659 (+ 940us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.998763 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":161,"threads_started":1}
I20260929 00:59:25.998905 14750 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: No bootstrap required, opened a new log
I20260929 00:59:25.999013 14750 ts_tablet_manager.cc:1397] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260929 00:59:25.999516 14750 raft_consensus.cc:348] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:25.999655 14750 raft_consensus.cc:374] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:25.999703 14750 raft_consensus.cc:729] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1e8b7eda924d0b86e34102a0f38d37, State: Initialized, Role: FOLLOWER
I20260929 00:59:25.999945 14750 consensus_queue.cc:260] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.000229 14750 ts_tablet_manager.cc:1428] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:59:26.001289 14930 tablet_service.cc:1467] Processing CreateTablet for tablet 630bd374aa0d41218f277293a1efcc14 (DEFAULT_TABLE table=test-table [id=0fca1e3f14fe44fbb1f8b0e294633f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:26.002460 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630bd374aa0d41218f277293a1efcc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:26.009042 15062 tablet_service.cc:1467] Processing CreateTablet for tablet 630bd374aa0d41218f277293a1efcc14 (DEFAULT_TABLE table=test-table [id=0fca1e3f14fe44fbb1f8b0e294633f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:26.009491 15141 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Bootstrap starting.
I20260929 00:59:26.010102 15062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630bd374aa0d41218f277293a1efcc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:26.011883 15141 tablet_bootstrap.cc:654] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:26.012606 15141 log.cc:824] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:26.014667 15141 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: No bootstrap required, opened a new log
I20260929 00:59:26.014864 15141 ts_tablet_manager.cc:1397] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:59:26.017023 15143 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Bootstrap starting.
I20260929 00:59:26.018143 15141 raft_consensus.cc:348] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.018450 15141 raft_consensus.cc:374] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:26.018498 15141 raft_consensus.cc:729] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa3c54cca2543849e1fdd9f955e8288, State: Initialized, Role: FOLLOWER
I20260929 00:59:26.018930 15141 consensus_queue.cc:260] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.019608 14996 heartbeater.cc:503] Master 127.0.98.62:40705 was elected leader, sending a full tablet report...
I20260929 00:59:26.019686 15143 tablet_bootstrap.cc:654] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:26.020008 15141 ts_tablet_manager.cc:1428] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:59:26.020547 15143 log.cc:824] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:26.022750 15143 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: No bootstrap required, opened a new log
I20260929 00:59:26.022964 15143 ts_tablet_manager.cc:1397] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:59:26.025938 15143 raft_consensus.cc:348] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.026149 15143 raft_consensus.cc:374] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:26.026248 15143 raft_consensus.cc:729] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ab389c33a34416a50b126e645d3057, State: Initialized, Role: FOLLOWER
I20260929 00:59:26.026615 15143 consensus_queue.cc:260] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.027168 15128 heartbeater.cc:503] Master 127.0.98.62:40705 was elected leader, sending a full tablet report...
I20260929 00:59:26.027405 15143 ts_tablet_manager.cc:1428] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260929 00:59:26.051182 14859 tablet.cc:2369] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:59:26.202098 15129 tablet.cc:2369] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:59:26.260800 14997 tablet.cc:2369] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:59:26.286746 15144 raft_consensus.cc:482] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:26.286998 15144 raft_consensus.cc:504] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.288316 15144 leader_election.cc:288] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165), 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163)
I20260929 00:59:26.293131 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:26.288945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40485 (local address 127.0.98.3:42165)
0929 00:59:26.289077 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:26.289081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:26.289096 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:59:26.289153 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:26.289156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:26.289204 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:26.289279 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:26.289687 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.290327 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.291848 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.292538 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.292651 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:26.292718 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:26.292951 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 00:59:26.293031 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:59:26.293150 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:26.288518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42165 (local address 127.0.98.2:40485)
0929 00:59:26.288860 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:26.288878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:26.288993 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:26.289393 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:26.289396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:26.289413 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:26.289550 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:26.289558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.290438 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.290444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:26.291716 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:26.291726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.292698 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.292701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:26.292795 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:26.292819 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:59:26.292895 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 00:59:26.292975 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260929 00:59:26.293447 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:26.288821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45195 (local address 127.0.98.1:34163)
0929 00:59:26.289238 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:26.289245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:26.289261 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:59:26.289315 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:26.289317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:26.289368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:26.289485 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:26.289852 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.290636 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.292031 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.292538 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.292698 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:26.292877 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:26.293212 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 00:59:26.293308 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":82,"threads_started":1}
I20260929 00:59:26.293469 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:26.288607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34163 (local address 127.0.98.2:45195)
0929 00:59:26.289074 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:26.289089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:26.289170 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:26.289495 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:26.289498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:26.289516 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:26.289705 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:26.289712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.290759 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.290763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:26.291890 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:26.291897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.292787 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.292791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:26.292900 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:26.292922 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:59:26.292989 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 00:59:26.293077 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":25,"thread_start_us":81,"threads_started":1}
I20260929 00:59:26.294253 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630bd374aa0d41218f277293a1efcc14" candidate_uuid: "1fa3c54cca2543849e1fdd9f955e8288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" is_pre_election: true
I20260929 00:59:26.294292 15082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630bd374aa0d41218f277293a1efcc14" candidate_uuid: "1fa3c54cca2543849e1fdd9f955e8288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ab389c33a34416a50b126e645d3057" is_pre_election: true
I20260929 00:59:26.294584 15082 raft_consensus.cc:2454] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa3c54cca2543849e1fdd9f955e8288 in term 0.
I20260929 00:59:26.294585 14812 raft_consensus.cc:2454] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa3c54cca2543849e1fdd9f955e8288 in term 0.
I20260929 00:59:26.295069 14885 leader_election.cc:302] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e1e8b7eda924d0b86e34102a0f38d37, 1fa3c54cca2543849e1fdd9f955e8288, 56ab389c33a34416a50b126e645d3057; no voters:
I20260929 00:59:26.295351 15144 raft_consensus.cc:2781] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:26.295433 15144 raft_consensus.cc:482] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:26.295490 15144 raft_consensus.cc:3037] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:26.296540 15144 raft_consensus.cc:504] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.296947 15144 leader_election.cc:288] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [CANDIDATE]: Term 1 election: Requested vote from peers 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165), 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163)
I20260929 00:59:26.297281 15082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630bd374aa0d41218f277293a1efcc14" candidate_uuid: "1fa3c54cca2543849e1fdd9f955e8288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ab389c33a34416a50b126e645d3057"
I20260929 00:59:26.297309 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630bd374aa0d41218f277293a1efcc14" candidate_uuid: "1fa3c54cca2543849e1fdd9f955e8288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1e8b7eda924d0b86e34102a0f38d37"
I20260929 00:59:26.297413 15082 raft_consensus.cc:3037] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:26.297422 14812 raft_consensus.cc:3037] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:26.298703 14812 raft_consensus.cc:2454] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa3c54cca2543849e1fdd9f955e8288 in term 1.
I20260929 00:59:26.298703 15082 raft_consensus.cc:2454] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa3c54cca2543849e1fdd9f955e8288 in term 1.
I20260929 00:59:26.298995 14886 leader_election.cc:302] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1e8b7eda924d0b86e34102a0f38d37, 1fa3c54cca2543849e1fdd9f955e8288; no voters:
I20260929 00:59:26.299181 15144 raft_consensus.cc:2781] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:26.299387 15144 raft_consensus.cc:686] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa3c54cca2543849e1fdd9f955e8288, State: Running, Role: LEADER
I20260929 00:59:26.299732 15144 consensus_queue.cc:237] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.303367 14669 catalog_manager.cc:5495] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1fa3c54cca2543849e1fdd9f955e8288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } health_report { overall_health: UNKNOWN } } }
I20260929 00:59:26.361538 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:26.358002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.0.98.2:44709)
0929 00:59:26.358144 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:26.358148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:26.358162 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:59:26.358367 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:26.358370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:26.358429 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:26.358522 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:26.358790 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.359282 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.360106 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:26.360310 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:26.360381 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:26.360658 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:26.361098 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:26.361188 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:26.361262 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 00:59:26.361358 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:59:26.367961 14812 raft_consensus.cc:1261] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Refusing update from remote peer 1fa3c54cca2543849e1fdd9f955e8288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:26.367962 15082 raft_consensus.cc:1261] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 1 FOLLOWER]: Refusing update from remote peer 1fa3c54cca2543849e1fdd9f955e8288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:26.368835 15150 consensus_queue.cc:1035] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:26.369086 15144 consensus_queue.cc:1035] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:26.377393 15153 mvcc.cc:205] Tried to move back new op lower bound from 7334476047834501120 to 7334476047566589952. Current Snapshot: MvccSnapshot[applied={T|T < 7334476047834501120}]
I20260929 00:59:26.384972 14668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:26.385121 14668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:26.387380 14668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Sending DeleteTablet for 3 replicas of tablet 630bd374aa0d41218f277293a1efcc14
I20260929 00:59:26.388055 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet 630bd374aa0d41218f277293a1efcc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32790
I20260929 00:59:26.388141 15062 tablet_service.cc:1514] Processing DeleteTablet for tablet 630bd374aa0d41218f277293a1efcc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49474
I20260929 00:59:26.388552 15163 tablet_replica.cc:330] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: stopping tablet replica
I20260929 00:59:26.388697 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet 630bd374aa0d41218f277293a1efcc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44102
I20260929 00:59:26.388886 15163 raft_consensus.cc:2229] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:59:26.389110 15165 tablet_replica.cc:330] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: stopping tablet replica
I20260929 00:59:26.389272 15163 raft_consensus.cc:2258] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:26.389333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 14708
I20260929 00:59:26.389286 15165 raft_consensus.cc:2229] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:26.389570 15165 raft_consensus.cc:2258] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:26.391971 15163 ts_tablet_manager.cc:1905] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:26.394392 15163 ts_tablet_manager.cc:1918] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:26.394498 15163 log.cc:1196] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/630bd374aa0d41218f277293a1efcc14
I20260929 00:59:26.394425 15164 tablet_replica.cc:330] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: stopping tablet replica
I20260929 00:59:26.394846 15163 ts_tablet_manager.cc:1939] T 630bd374aa0d41218f277293a1efcc14 P 1fa3c54cca2543849e1fdd9f955e8288: Deleting consensus metadata
I20260929 00:59:26.395016 15164 raft_consensus.cc:2229] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:26.395373 15164 raft_consensus.cc:2258] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:26.396405 14654 catalog_manager.cc:4841] TS 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709): tablet 630bd374aa0d41218f277293a1efcc14 (table test-table [id=0fca1e3f14fe44fbb1f8b0e294633f80]) successfully deleted
I20260929 00:59:26.396859 15164 ts_tablet_manager.cc:1905] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 00:59:26.396859 14657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34163 (error 108)
W20260929 00:59:26.398073 15138 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:26.397757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34163 (local address 127.0.0.1:44106)
0929 00:59:26.397902 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:26.397976 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34163: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 00:59:26.398262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 14863
I20260929 00:59:26.399631 15164 ts_tablet_manager.cc:1918] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:26.399760 15164 log.cc:1196] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/630bd374aa0d41218f277293a1efcc14
W20260929 00:59:26.399698 14657 catalog_manager.cc:4584] TS 0e1e8b7eda924d0b86e34102a0f38d37 (127.0.98.1:34163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 630bd374aa0d41218f277293a1efcc14: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34163: connect: Connection reset by peer (error 104)
I20260929 00:59:26.400112 15164 ts_tablet_manager.cc:1939] T 630bd374aa0d41218f277293a1efcc14 P 56ab389c33a34416a50b126e645d3057: Deleting consensus metadata
I20260929 00:59:26.401168 14656 catalog_manager.cc:4841] TS 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165): tablet 630bd374aa0d41218f277293a1efcc14 (table test-table [id=0fca1e3f14fe44fbb1f8b0e294633f80]) successfully deleted
I20260929 00:59:26.404928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 14999
I20260929 00:59:26.410467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 14637
I20260929 00:59:26.416597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40705
--webserver_interface=127.0.98.62
--webserver_port=46045
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:26.526895 15167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:26.527155 15167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:26.527242 15167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:26.531363 15167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:26.531421 15167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:26.531482 15167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:26.531520 15167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:26.531554 15167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:26.536782 15167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:26.537818 15167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:26.538865 15167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:26.546911 15172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:26.546933 15173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:26.546948 15175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:26.547295 15167 server_base.cc:1034] running on GCE node
I20260929 00:59:26.547777 15167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:26.548794 15167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:26.549952 15167 hybrid_clock.cc:648] HybridClock initialized: now 1790643566549937 us; error 26 us; skew 500 ppm
Sep 29 00:59:26 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:26.552628 15167 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:59:26.553491 15167 webserver.cc:460] Webserver started at http://127.0.98.62:46045/ using document root <none> and password file <none>
I20260929 00:59:26.554018 15167 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:26.554109 15167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:26.557672 15167 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:59:26.559677 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:26.560786 15167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:59:26.560917 15167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "0938835f982e4bff9bd647ed55764c80"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "99c67bc080307abb5134b0bfcecd358c"
server_key_iv: "52fbd9388d395d020f9f2645f3560af9"
server_key_version: "encryptionkey@0"
I20260929 00:59:26.561424 15167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:26.580377 15167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:26.583137 15167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:26.583344 15167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:26.590129 15167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40705
I20260929 00:59:26.590420 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40705 every 8 connection(s)
I20260929 00:59:26.591251 15167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:26.592346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 15167
I20260929 00:59:26.592831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34163
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40705
--webserver_port=42741
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:26.596357 15235 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:59:26.597913 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Bootstrap starting.
I20260929 00:59:26.615419 15235 log.cc:824] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:26.626487 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:59:26.626920 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80: Bootstrap complete.
I20260929 00:59:26.631060 15235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:26.631582 15235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0938835f982e4bff9bd647ed55764c80, State: Initialized, Role: FOLLOWER
I20260929 00:59:26.632073 15235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:26.632185 15235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:26.632283 15235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:26.632385 15235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:59:26.634379 15235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:26.634708 15235 leader_election.cc:302] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0938835f982e4bff9bd647ed55764c80; no voters:
I20260929 00:59:26.635026 15235 leader_election.cc:288] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 00:59:26.635175 15239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 00:59:26.635465 15239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [term 3 LEADER]: Becoming Leader. State: Replica: 0938835f982e4bff9bd647ed55764c80, State: Running, Role: LEADER
I20260929 00:59:26.635828 15239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260929 00:59:26.636293 15235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:26.636778 15239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0938835f982e4bff9bd647ed55764c80. Latest consensus state: current_term: 3 leader_uuid: "0938835f982e4bff9bd647ed55764c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } } }
I20260929 00:59:26.636891 15239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:26.637226 15243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:26.637447 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0938835f982e4bff9bd647ed55764c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0938835f982e4bff9bd647ed55764c80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } } }
I20260929 00:59:26.637622 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0938835f982e4bff9bd647ed55764c80 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:26.642108 15243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:26.642771 15243 catalog_manager.cc:1259] Loaded cluster ID: 2a5c652890ac4eac80bcb8218344b5ea
I20260929 00:59:26.642834 15243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 00:59:26.643790 15243 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb5b330a9ad google::LogMessage::Fail() at ??:0
@ 0x7fb5b330d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb5b330a4de google::LogMessage::Flush() at ??:0
@ 0x7fb5b330ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb5bdc49fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb5bdc4ad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb5bdc47579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb5bdc8b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb5b555c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb5b555cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb5b555e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb5b554d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb5b6b916db start_thread at ??:0
@ 0x7fb5b1e0861f clone at ??:0
W20260929 00:59:26.738088 15237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:26.738435 15237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:26.738507 15237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:26.742616 15237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:26.742671 15237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:26.742791 15237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:59:26.748077 15237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42741
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:26.749089 15237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:26.750075 15237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:26.759613 15276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:26.759639 15277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:26.759658 15279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:26.759814 15237 server_base.cc:1034] running on GCE node
I20260929 00:59:26.760277 15237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:26.760963 15237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:26.762144 15237 hybrid_clock.cc:648] HybridClock initialized: now 1790643566762115 us; error 45 us; skew 500 ppm
Sep 29 00:59:26 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:26.764995 15237 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:59:26.765822 15237 webserver.cc:460] Webserver started at http://127.0.98.1:42741/ using document root <none> and password file <none>
I20260929 00:59:26.766436 15237 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:26.766528 15237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:26.770579 15237 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:26.772809 15286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:26.774107 15237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:26.774255 15237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "0e1e8b7eda924d0b86e34102a0f38d37"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "080c9846f1806b09e2ae9b517dd877b7"
server_key_iv: "091af60ae198bf959fcb6b3886d3795a"
server_key_version: "encryptionkey@0"
I20260929 00:59:26.774827 15237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:26.804556 15237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:26.807925 15237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:26.808128 15237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:26.808748 15237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:26.809887 15293 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 00:59:26.815487 15237 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 00:59:26.815569 15237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 00:59:26.815650 15237 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 00:59:26.817359 15237 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 00:59:26.817430 15237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 00:59:26.817731 15293 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Bootstrap starting.
I20260929 00:59:26.828150 15237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34163
I20260929 00:59:26.829094 15237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:26.829653 15400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34163 every 8 connection(s)
I20260929 00:59:26.830827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 15237
I20260929 00:59:26.831282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44709
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40705
--webserver_port=39809
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:26.839762 15293 log.cc:824] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:26.849261 15293 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:59:26.849756 15293 tablet_bootstrap.cc:492] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Bootstrap complete.
I20260929 00:59:26.851248 15293 ts_tablet_manager.cc:1397] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Time spent bootstrapping tablet: real 0.034s user 0.028s sys 0.003s
I20260929 00:59:26.854590 15293 raft_consensus.cc:348] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.855280 15293 raft_consensus.cc:729] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1e8b7eda924d0b86e34102a0f38d37, State: Initialized, Role: FOLLOWER
I20260929 00:59:26.855888 15293 consensus_queue.cc:260] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:26.857080 15293 ts_tablet_manager.cc:1428] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 00:59:26.881712 15403 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:26.831286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:52083)
0929 00:59:26.832463 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:26.832521 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:26.833727 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:26.880756 (+ 47029us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":145,"threads_started":1}
W20260929 00:59:26.882480 15401 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40705 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40705: Client connection negotiation failed: client connection to 127.0.98.62:40705: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 00:59:26.883356 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:26.883009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:37051)
0929 00:59:26.883173 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:26.883257 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:59:26.884235 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:26.883970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:49437)
0929 00:59:26.884103 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:26.884163 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 00:59:26.884496 15401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:59:26.959868 15404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:26.960147 15404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:26.960230 15404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:26.964247 15404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:26.964330 15404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:26.964447 15404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:59:26.969645 15404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39809
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:26.970772 15404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:26.971784 15404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:26.981712 15411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:26.981779 15412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:26.981714 15414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:26.981982 15404 server_base.cc:1034] running on GCE node
I20260929 00:59:26.982436 15404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:26.983071 15404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:26.984228 15404 hybrid_clock.cc:648] HybridClock initialized: now 1790643566984197 us; error 44 us; skew 500 ppm
Sep 29 00:59:26 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:26.986913 15404 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:59:26.987737 15404 webserver.cc:460] Webserver started at http://127.0.98.2:39809/ using document root <none> and password file <none>
I20260929 00:59:26.988282 15404 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:26.988344 15404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:26.992355 15404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:26.994428 15421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:26.995605 15404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:59:26.995733 15404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "1fa3c54cca2543849e1fdd9f955e8288"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "8023423c1ad0432107dfed8402677cc9"
server_key_iv: "cda90387398150d22cf82baa326fc625"
server_key_version: "encryptionkey@0"
I20260929 00:59:26.996263 15404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:27.008459 15404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:27.011339 15404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:27.011545 15404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:27.012132 15404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:27.012948 15404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:27.012995 15404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:27.013042 15404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:27.013098 15404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:27.021703 15404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44709
I20260929 00:59:27.021792 15534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44709 every 8 connection(s)
I20260929 00:59:27.022780 15404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:27.026306 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.024804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:33987)
0929 00:59:27.025455 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:27.025559 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":110,"threads_started":1}
W20260929 00:59:27.026790 15535 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40705 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40705: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
I20260929 00:59:27.027315 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.027119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:41561)
0929 00:59:27.027223 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:27.027267 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 00:59:27.027966 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.027754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:54041)
0929 00:59:27.027859 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:27.027906 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 00:59:27.028127 15535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:59:27.031267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 15404
I20260929 00:59:27.031620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42165
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40705
--webserver_port=41049
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:27.141908 15538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:27.142261 15538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:27.142378 15538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:27.146373 15538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:27.146466 15538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:27.146561 15538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:59:27.151698 15538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41049
--tserver_master_addrs=127.0.98.62:40705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:27.152750 15538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:27.153762 15538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:27.163009 15544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:27.163017 15543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:27.163023 15546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:27.163127 15538 server_base.cc:1034] running on GCE node
I20260929 00:59:27.163611 15538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:27.164211 15538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:27.165369 15538 hybrid_clock.cc:648] HybridClock initialized: now 1790643567165352 us; error 29 us; skew 500 ppm
Sep 29 00:59:27 dist-test-slave-30s6 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:27.168147 15538 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:59:27.168987 15538 webserver.cc:460] Webserver started at http://127.0.98.3:41049/ using document root <none> and password file <none>
I20260929 00:59:27.169534 15538 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:27.169595 15538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:27.173228 15538 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:27.175209 15553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:27.176376 15538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:59:27.176514 15538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "56ab389c33a34416a50b126e645d3057"
format_stamp: "Formatted at 2026-09-29 00:59:24 on dist-test-slave-30s6"
server_key: "5f1f82c17f90713f8c04aeb4c84b67b4"
server_key_iv: "e7e68d61a302582a5c396aea425aaa54"
server_key_version: "encryptionkey@0"
I20260929 00:59:27.177083 15538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:27.192258 15538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:27.195513 15538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:27.195709 15538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:27.196293 15538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:27.197081 15538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:27.197151 15538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:27.197209 15538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:27.197263 15538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:27.206615 15538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42165
I20260929 00:59:27.206651 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42165 every 8 connection(s)
I20260929 00:59:27.207600 15538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:27.211408 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.209700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:49795)
0929 00:59:27.210148 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:27.210692 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":101,"threads_started":1}
W20260929 00:59:27.211870 15667 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40705 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40705: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
I20260929 00:59:27.212440 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.212242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:44823)
0929 00:59:27.212341 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:27.212387 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 00:59:27.213025 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.212840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:54707)
0929 00:59:27.212929 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:27.212978 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 00:59:27.213204 15667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:59:27.217690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 15538
I20260929 00:59:27.885426 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.885023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:50277)
0929 00:59:27.885249 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:27.885315 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260929 00:59:28.029006 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:28.028633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:58577)
0929 00:59:28.028863 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:28.028915 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260929 00:59:28.214270 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:28.213792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:39907)
0929 00:59:28.214055 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:28.214109 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260929 00:59:28.686690 15674 raft_consensus.cc:482] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:28.687021 15674 raft_consensus.cc:504] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:28.688081 15674 leader_election.cc:288] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709), 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165)
Sep 29 00:59:28 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 29 00:59:28 dist-test-slave-30s6 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260929 00:59:28.701849 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:28.688571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47701 (local address 127.0.98.2:44709)
0929 00:59:28.688888 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:28.688903 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:28.689120 (+ 217us) server_negotiation.cc:407] Connection header received
0929 00:59:28.689825 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:28.689831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:28.690134 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:28.690484 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:28.691649 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:28.692696 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:28.693851 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:28.694393 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:28.694547 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:28.697200 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:28.697207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:28.697225 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:28.697265 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:28.699067 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:28.699738 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:28.699746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:28.699753 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:28.699928 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:28.700372 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:28.700375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:28.700376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:28.700751 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:28.700889 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:28.701417 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 00:59:28.701669 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260929 00:59:28.701838 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:28.688344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44709 (local address 127.0.98.1:47701)
0929 00:59:28.688996 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:28.689009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:28.689111 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:28.690843 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:28.690845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:28.691075 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:28.691477 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:28.691493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:28.692860 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:28.692863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:28.693733 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:28.693742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:28.694516 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:28.694520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:28.694734 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:28.695379 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:28.695395 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:28.696953 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:28.699204 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:28.699210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:28.699224 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:28.699587 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:28.700120 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:28.700123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:28.700125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:28.700284 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:28.700880 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:28.700886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:28.701112 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 00:59:28.701292 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 00:59:28.701636 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":241,"spinlock_wait_cycles":19712,"thread_start_us":68,"threads_started":1}
I20260929 00:59:28.701894 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:28.688563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43657 (local address 127.0.98.3:42165)
0929 00:59:28.688976 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:28.689011 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:28.689048 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:59:28.689790 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:28.689796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:28.690134 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:28.690466 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:28.691685 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:28.692684 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:28.693896 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:28.694412 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:28.694584 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:28.697485 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:28.697490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:28.697500 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:28.697525 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:28.699625 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:28.700121 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:28.700127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:28.700132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:28.700296 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:28.700595 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:28.700598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:28.700600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:28.700965 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:28.701098 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:28.701431 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 00:59:28.701706 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":97,"threads_started":1}
I20260929 00:59:28.701885 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:28.688344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42165 (local address 127.0.98.1:43657)
0929 00:59:28.688637 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:28.688673 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:28.688781 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:28.690543 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:28.690549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:28.691046 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:28.691477 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:28.691495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:28.692816 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:28.692820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:28.693740 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:28.693749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:28.694553 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:28.694556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:28.694739 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:28.695434 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:28.695446 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:28.697311 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:28.699766 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:28.699769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:28.699771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:28.700016 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:28.700406 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:28.700409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:28.700411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:28.700497 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:28.701080 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:28.701083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:28.701176 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 00:59:28.701293 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 00:59:28.701726 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":268,"spinlock_wait_cycles":5888,"thread_start_us":59,"threads_started":1}
W20260929 00:59:28.705055 15289 leader_election.cc:341] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165): Not found: Tablet not found: 630bd374aa0d41218f277293a1efcc14
W20260929 00:59:28.705229 15287 leader_election.cc:341] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709): Not found: Tablet not found: 630bd374aa0d41218f277293a1efcc14
I20260929 00:59:28.705330 15287 leader_election.cc:302] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e1e8b7eda924d0b86e34102a0f38d37; no voters: 1fa3c54cca2543849e1fdd9f955e8288, 56ab389c33a34416a50b126e645d3057
I20260929 00:59:28.705578 15674 raft_consensus.cc:2726] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:59:28.886440 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:28.886117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:59415)
0929 00:59:28.886302 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:28.886352 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 00:59:29.030134 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:29.029711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:59571)
0929 00:59:29.029966 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:29.030027 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260929 00:59:29.215256 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:29.214919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:46627)
0929 00:59:29.215126 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:29.215177 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260929 00:59:29.887643 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:29.887226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:42821)
0929 00:59:29.887486 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:29.887548 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260929 00:59:30.031330 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:30.030917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:47461)
0929 00:59:30.031166 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:30.031240 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260929 00:59:30.216382 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:30.215978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:48477)
0929 00:59:30.216221 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:30.216280 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260929 00:59:30.775765 15684 raft_consensus.cc:482] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:30.775892 15684 raft_consensus.cc:504] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:30.776403 15684 leader_election.cc:288] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709), 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165)
W20260929 00:59:30.777213 15287 leader_election.cc:341] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709): Not found: Tablet not found: 630bd374aa0d41218f277293a1efcc14
W20260929 00:59:30.777348 15289 leader_election.cc:341] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165): Not found: Tablet not found: 630bd374aa0d41218f277293a1efcc14
I20260929 00:59:30.777460 15289 leader_election.cc:302] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e1e8b7eda924d0b86e34102a0f38d37; no voters: 1fa3c54cca2543849e1fdd9f955e8288, 56ab389c33a34416a50b126e645d3057
I20260929 00:59:30.777700 15684 raft_consensus.cc:2726] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:59:30.888852 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:30.888442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:49583)
0929 00:59:30.888697 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:30.888755 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260929 00:59:31.032481 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:31.032082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:36207)
0929 00:59:31.032334 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:31.032392 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260929 00:59:31.217376 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:31.217025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:56683)
0929 00:59:31.217248 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:31.217295 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260929 00:59:31.890085 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:31.889674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:43235)
0929 00:59:31.889936 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:31.889991 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260929 00:59:32.033677 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:32.033216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:41761)
0929 00:59:32.033527 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:32.033583 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":126,"threads_started":1}
I20260929 00:59:32.218719 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:32.218248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:43159)
0929 00:59:32.218549 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:32.218618 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260929 00:59:32.891647 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:32.891051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:40035)
0929 00:59:32.891472 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:32.891542 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":172,"threads_started":1}
I20260929 00:59:33.035012 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:33.034565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:41875)
0929 00:59:33.034844 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:33.034912 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260929 00:59:33.224769 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:33.224228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:34807)
0929 00:59:33.224550 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:33.224620 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260929 00:59:33.893013 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:33.892589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:42251)
0929 00:59:33.892853 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:33.892911 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260929 00:59:34.036260 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:34.035746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:49561)
0929 00:59:34.036072 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:34.036133 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":75,"threads_started":1}
I20260929 00:59:34.112476 15696 raft_consensus.cc:482] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:34.112635 15696 raft_consensus.cc:504] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:34.113114 15696 leader_election.cc:288] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709), 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165)
W20260929 00:59:34.113946 15287 leader_election.cc:341] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709): Not found: Tablet not found: 630bd374aa0d41218f277293a1efcc14
W20260929 00:59:34.114091 15289 leader_election.cc:341] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165): Not found: Tablet not found: 630bd374aa0d41218f277293a1efcc14
I20260929 00:59:34.114228 15289 leader_election.cc:302] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e1e8b7eda924d0b86e34102a0f38d37; no voters: 1fa3c54cca2543849e1fdd9f955e8288, 56ab389c33a34416a50b126e645d3057
I20260929 00:59:34.114404 15696 raft_consensus.cc:2726] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:59:34.226009 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:34.225560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:55731)
0929 00:59:34.225849 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:34.225916 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":71,"threads_started":1}
I20260929 00:59:34.894164 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:34.893762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:47245)
0929 00:59:34.894005 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:34.894066 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260929 00:59:35.037468 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:35.037067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:41795)
0929 00:59:35.037318 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:35.037373 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260929 00:59:35.227152 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:35.226726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:39707)
0929 00:59:35.227003 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:35.227058 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20260929 00:59:35.895555 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:35.895032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:50101)
0929 00:59:35.895360 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:35.895443 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":90,"threads_started":1}
I20260929 00:59:36.038961 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:36.038476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:54611)
0929 00:59:36.038757 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:36.038835 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":126,"threads_started":1}
I20260929 00:59:36.228504 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:36.228055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:52071)
0929 00:59:36.228325 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:36.228401 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260929 00:59:36.896945 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:36.896514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:39341)
0929 00:59:36.896781 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:36.896847 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260929 00:59:37.040189 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:37.039765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:42485)
0929 00:59:37.040031 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:37.040090 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260929 00:59:37.229632 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:37.229252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:35841)
0929 00:59:37.229484 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:37.229535 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260929 00:59:37.329905 15707 raft_consensus.cc:482] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:37.330075 15707 raft_consensus.cc:504] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa3c54cca2543849e1fdd9f955e8288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44709 } } peers { permanent_uuid: "56ab389c33a34416a50b126e645d3057" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42165 } } peers { permanent_uuid: "0e1e8b7eda924d0b86e34102a0f38d37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34163 } }
I20260929 00:59:37.330595 15707 leader_election.cc:288] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709), 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165)
W20260929 00:59:37.331328 15287 leader_election.cc:341] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fa3c54cca2543849e1fdd9f955e8288 (127.0.98.2:44709): Not found: Tablet not found: 630bd374aa0d41218f277293a1efcc14
W20260929 00:59:37.331420 15289 leader_election.cc:341] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ab389c33a34416a50b126e645d3057 (127.0.98.3:42165): Not found: Tablet not found: 630bd374aa0d41218f277293a1efcc14
I20260929 00:59:37.331463 15287 leader_election.cc:302] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e1e8b7eda924d0b86e34102a0f38d37; no voters: 1fa3c54cca2543849e1fdd9f955e8288, 56ab389c33a34416a50b126e645d3057
I20260929 00:59:37.331601 15707 raft_consensus.cc:2726] T 630bd374aa0d41218f277293a1efcc14 P 0e1e8b7eda924d0b86e34102a0f38d37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:59:37.898389 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:37.897872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:49509)
0929 00:59:37.898137 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:37.898255 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260929 00:59:38.041471 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:38.041043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:44489)
0929 00:59:38.041309 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:38.041376 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260929 00:59:38.230868 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:38.230481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:42607)
0929 00:59:38.230722 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:38.230776 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260929 00:59:38.899559 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:38.899118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:48971)
0929 00:59:38.899397 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:38.899464 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
I20260929 00:59:39.042768 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:39.042326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:40223)
0929 00:59:39.042584 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:39.042662 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260929 00:59:39.232071 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:39.231553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:58597)
0929 00:59:39.231844 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:39.231946 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260929 00:59:39.900725 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:39.900307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:47481)
0929 00:59:39.900572 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:39.900626 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260929 00:59:40.043905 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:40.043524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:53137)
0929 00:59:40.043758 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:40.043811 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260929 00:59:40.233295 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:40.232846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:43909)
0929 00:59:40.233128 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:40.233191 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260929 00:59:40.901978 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:40.901585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:41423)
0929 00:59:40.901825 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:40.901884 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260929 00:59:41.045104 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:41.044683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:40853)
0929 00:59:41.044957 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:41.045010 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":109,"threads_started":1}
I20260929 00:59:41.234472 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:41.234041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.3:35083)
0929 00:59:41.234327 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:41.234377 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260929 00:59:41.903378 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:41.902943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.1:38137)
0929 00:59:41.903207 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:41.903275 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260929 00:59:42.046396 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:42.045957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40705 (local address 127.0.98.2:33381)
0929 00:59:42.046243 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:42.046298 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260929 00:59:42.219427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 15237
I20260929 00:59:42.225015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 15404
I20260929 00:59:42.230276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 15538
2026-09-29T00:59:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19469 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:42 dist-test-slave-30s6 krb5kdc[15724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:42 dist-test-slave-30s6 krb5kdc[15724](info): set up 2 sockets
Sep 29 00:59:42 dist-test-slave-30s6 krb5kdc[15724](info): commencing operation
krb5kdc: starting...
W20260929 00:59:43.278383 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 00:59:43 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 00:59:43.400354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39987
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:43.511148 15737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:43.511401 15737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:43.511471 15737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:43.515517 15737 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 00:59:43.515568 15737 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 00:59:43.515583 15737 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 00:59:43.515597 15737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:43.515648 15737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:43.515681 15737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:43.515709 15737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:43.515758 15737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:43.520915 15737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:43.521967 15737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:43.523034 15737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:43.531507 15743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:43.531507 15745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:43.531590 15737 server_base.cc:1034] running on GCE node
W20260929 00:59:43.531507 15742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:43.532281 15737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:43.533371 15737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:43.534523 15737 hybrid_clock.cc:648] HybridClock initialized: now 1790643583534505 us; error 31 us; skew 500 ppm
Sep 29 00:59:43 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:43.537218 15737 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 00:59:43.538053 15737 webserver.cc:460] Webserver started at http://127.0.98.62:37341/ using document root <none> and password file <none>
I20260929 00:59:43.538640 15737 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:43.538702 15737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:43.538926 15737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:43.540910 15737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "45f3d8faa5614970bbb3d8c78f009511"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "8c0333661b6ba850b7c02dec58e4bd17"
server_key_iv: "6372fba098c307bb944630779c508e67"
server_key_version: "encryptionkey@0"
I20260929 00:59:43.541466 15737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "45f3d8faa5614970bbb3d8c78f009511"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "8c0333661b6ba850b7c02dec58e4bd17"
server_key_iv: "6372fba098c307bb944630779c508e67"
server_key_version: "encryptionkey@0"
I20260929 00:59:43.544993 15737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 00:59:43.547349 15752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:43.548450 15737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:59:43.548583 15737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "45f3d8faa5614970bbb3d8c78f009511"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "8c0333661b6ba850b7c02dec58e4bd17"
server_key_iv: "6372fba098c307bb944630779c508e67"
server_key_version: "encryptionkey@0"
I20260929 00:59:43.548681 15737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:43.559433 15737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:43.562069 15737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:43.562278 15737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:43.569049 15737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39987
I20260929 00:59:43.569337 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39987 every 8 connection(s)
I20260929 00:59:43.569969 15737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:43.572736 15805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:43.576788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 15737
I20260929 00:59:43.576895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 00:59:43.577175 392 external_mini_cluster.cc:1428] Setting key a629194c3141827a9dea07c672ce973d
I20260929 00:59:43.578063 15805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Bootstrap starting.
I20260929 00:59:43.580379 15805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:43.581250 15805 log.cc:824] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:43.583299 15805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: No bootstrap required, opened a new log
Sep 29 00:59:43 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:43.587591 15805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 00:59:43.587834 15805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:43.587914 15805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f3d8faa5614970bbb3d8c78f009511, State: Initialized, Role: FOLLOWER
I20260929 00:59:43.588321 15805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 00:59:43.588470 15805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:43.588547 15805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:43.588656 15805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:43.589463 15805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 00:59:43.589781 15805 leader_election.cc:302] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f3d8faa5614970bbb3d8c78f009511; no voters:
I20260929 00:59:43.590093 15805 leader_election.cc:288] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:43.590194 15810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:43.590415 15810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 1 LEADER]: Becoming Leader. State: Replica: 45f3d8faa5614970bbb3d8c78f009511, State: Running, Role: LEADER
I20260929 00:59:43.590775 15810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 00:59:43.591246 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:43.578472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.98.62:39987)
0929 00:59:43.578851 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:43.578866 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:43.578899 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:59:43.579499 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:43.579514 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:43.579831 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:43.580102 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:43.580986 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:43.581997 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:43.583131 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:43.583624 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:43.583739 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:43.586389 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:43.586394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:43.586407 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:43.586432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:43.588272 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:43.588743 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:43.588749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:43.588756 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:43.588881 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:43.589156 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:43.589159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:43.589160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:43.589528 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:43.589772 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:43.590039 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 00:59:43.590373 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":115,"threads_started":1}
I20260929 00:59:43.591457 15805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:43.592461 15811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f3d8faa5614970bbb3d8c78f009511. Latest consensus state: current_term: 1 leader_uuid: "45f3d8faa5614970bbb3d8c78f009511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } } }
I20260929 00:59:43.592586 15811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:43.592871 15812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f3d8faa5614970bbb3d8c78f009511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } } }
I20260929 00:59:43.592945 15812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:43.593402 15819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:43.596119 15819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:43.600973 15819 catalog_manager.cc:1347] Generated new cluster ID: dede7f9ec350415abcd2021afdcea247
I20260929 00:59:43.601080 15819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:43.622344 15819 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:43.623632 15819 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:43.630127 15819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Generated new TSK 0
I20260929 00:59:43.630971 15819 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 00:59:43.657114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39987
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:43.769905 15831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:43.770169 15831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:43.770282 15831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:43.774240 15831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:43.774290 15831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:43.774380 15831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 00:59:43.779454 15831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:43.780435 15831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:43.781422 15831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:43.790460 15836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:43.790544 15839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:43.790671 15831 server_base.cc:1034] running on GCE node
W20260929 00:59:43.790558 15837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:43.791145 15831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:43.791783 15831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:43.792969 15831 hybrid_clock.cc:648] HybridClock initialized: now 1790643583792957 us; error 29 us; skew 500 ppm
Sep 29 00:59:43 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:43.795656 15831 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 00:59:43.796482 15831 webserver.cc:460] Webserver started at http://127.0.98.1:33347/ using document root <none> and password file <none>
I20260929 00:59:43.796984 15831 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:43.797040 15831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:43.797204 15831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:43.799160 15831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "ca028b96c255482b9b6f3d044a4aa893"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "5217746c931a990d52cb3277cf6c77ae"
server_key_iv: "742ad1b84cf3de61c5144d68bb452f94"
server_key_version: "encryptionkey@0"
I20260929 00:59:43.799649 15831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ca028b96c255482b9b6f3d044a4aa893"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "5217746c931a990d52cb3277cf6c77ae"
server_key_iv: "742ad1b84cf3de61c5144d68bb452f94"
server_key_version: "encryptionkey@0"
I20260929 00:59:43.803201 15831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:43.805560 15846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:43.806792 15831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:43.806914 15831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "ca028b96c255482b9b6f3d044a4aa893"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "5217746c931a990d52cb3277cf6c77ae"
server_key_iv: "742ad1b84cf3de61c5144d68bb452f94"
server_key_version: "encryptionkey@0"
I20260929 00:59:43.807022 15831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:43.823634 15831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:43.826503 15831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:43.826701 15831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:43.827299 15831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:43.828140 15831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:43.828207 15831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:43.828275 15831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:43.828322 15831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:43.837109 15831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34505
I20260929 00:59:43.837131 15959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34505 every 8 connection(s)
I20260929 00:59:43.838003 15831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:43.842962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 15831
I20260929 00:59:43.843094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 00:59:43.843459 392 external_mini_cluster.cc:1428] Setting key 783d5e46b930b32778e1185de5465d84
Sep 29 00:59:43 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:43.852537 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:43.839884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51195 (local address 127.0.98.62:39987)
0929 00:59:43.840022 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:43.840026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:43.841019 (+ 993us) server_negotiation.cc:407] Connection header received
0929 00:59:43.841835 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:43.841837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:43.841883 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:43.841958 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:43.843337 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:43.844061 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:43.845387 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:43.845615 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:43.845782 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:43.848547 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:43.848554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:43.848560 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:43.848593 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:43.850351 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:43.850866 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:43.850872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:43.850874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:43.850975 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:43.851301 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:43.851305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:43.851309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:43.851571 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:43.851675 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:43.852222 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 00:59:43.852350 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:59:43.853330 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:43.840251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:51195)
0929 00:59:43.840877 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:43.840919 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:43.841662 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:43.842115 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:43.842121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:43.842505 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:43.843160 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:43.843187 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:43.844198 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:43.844201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:43.845254 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:43.845268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:43.845837 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:43.845844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:43.846084 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:43.846866 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:43.846882 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:43.848324 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:43.850474 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:43.850479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:43.850491 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:43.850763 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:43.851066 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:43.851068 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:43.851069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:43.851202 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:43.851680 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:43.851685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:43.851916 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 00:59:43.852164 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 00:59:43.852429 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 00:59:43.854512 15960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39987
I20260929 00:59:43.854763 15960 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:43.855319 15960 heartbeater.cc:511] Master 127.0.98.62:39987 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 00:59:43.856871 15769 ts_manager.cc:194] Registered new tserver with Master: ca028b96c255482b9b6f3d044a4aa893 (127.0.98.1:34505)
I20260929 00:59:43.857978 15769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 00:59:43.870712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39987
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:43.978927 15965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:43.979177 15965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:43.979219 15965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:43.983119 15965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:43.983173 15965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:43.983230 15965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 00:59:43.988283 15965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:43.989236 15965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:43.990224 15965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:43.999966 15973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:44.000010 15971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:44.000195 15965 server_base.cc:1034] running on GCE node
W20260929 00:59:44.000103 15970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:44.000684 15965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:44.001334 15965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:44.002517 15965 hybrid_clock.cc:648] HybridClock initialized: now 1790643584002499 us; error 30 us; skew 500 ppm
Sep 29 00:59:43 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:44.005318 15965 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 00:59:44.006294 15965 webserver.cc:460] Webserver started at http://127.0.98.2:34403/ using document root <none> and password file <none>
I20260929 00:59:44.006863 15965 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:44.006951 15965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:44.007176 15965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:44.009253 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba27d79c3b524ac48bd9d176ba54b614"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "29c2bf04f011bca2c94697ec2cd1148a"
server_key_iv: "03d8281b4222a2b40bc69f04758bec52"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.009810 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba27d79c3b524ac48bd9d176ba54b614"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "29c2bf04f011bca2c94697ec2cd1148a"
server_key_iv: "03d8281b4222a2b40bc69f04758bec52"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.013643 15965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:59:44.016062 15980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:44.017201 15965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:44.017343 15965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "ba27d79c3b524ac48bd9d176ba54b614"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "29c2bf04f011bca2c94697ec2cd1148a"
server_key_iv: "03d8281b4222a2b40bc69f04758bec52"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.017455 15965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:44.029264 15965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:44.032140 15965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:44.032351 15965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:44.032969 15965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:44.033830 15965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:44.033910 15965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:44.033981 15965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:44.034020 15965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:44.042665 15965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33873
I20260929 00:59:44.042688 16093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33873 every 8 connection(s)
I20260929 00:59:44.043532 15965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:44.046501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 15965
I20260929 00:59:44.046631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 00:59:44.046963 392 external_mini_cluster.cc:1428] Setting key 03e8952eda3b9688e36cbdc606fb3ea0
Sep 29 00:59:44 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 00:59:44.057943 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.045237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56495 (local address 127.0.98.62:39987)
0929 00:59:44.045402 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.045407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.046537 (+ 1130us) server_negotiation.cc:407] Connection header received
0929 00:59:44.047724 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.047728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.047795 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.047874 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:44.049398 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.049950 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.051182 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.051596 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.051717 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.054207 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:44.054211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:44.054215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:44.054243 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:44.055897 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:44.056382 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:44.056386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:44.056388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:44.056479 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:44.056814 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:44.056820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:44.056821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:44.057019 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:44.057123 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.057647 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.057776 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:59:44.058921 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.045658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:56495)
0929 00:59:44.046377 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.046437 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:44.047460 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:44.048027 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:44.048035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:44.048526 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:44.049233 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.049257 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.050120 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.050126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.051077 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.051087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.051746 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.051749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.051889 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.052566 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:44.052584 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:44.054013 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:44.056013 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:44.056018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:44.056031 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:44.056282 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:44.056570 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:44.056572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:44.056574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:44.056699 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:44.057145 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:44.057153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:44.057406 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 00:59:44.057629 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 00:59:44.057983 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 00:59:44.060173 16094 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39987
I20260929 00:59:44.060444 16094 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:44.061022 16094 heartbeater.cc:511] Master 127.0.98.62:39987 requested a full tablet report, sending...
I20260929 00:59:44.062229 15769 ts_manager.cc:194] Registered new tserver with Master: ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873)
I20260929 00:59:44.063238 15769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 00:59:44.077343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39987
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:44.187085 16099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:44.187367 16099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:44.187408 16099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:44.191414 16099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:44.191475 16099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:44.191571 16099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 00:59:44.197543 16099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:59:44.198637 16099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:44.199620 16099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:44.208846 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:44.208887 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:44.208904 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:44.209133 16099 server_base.cc:1034] running on GCE node
I20260929 00:59:44.209606 16099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:44.210353 16099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:44.211546 16099 hybrid_clock.cc:648] HybridClock initialized: now 1790643584211535 us; error 32 us; skew 500 ppm
Sep 29 00:59:44 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:44.214270 16099 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 00:59:44.215139 16099 webserver.cc:460] Webserver started at http://127.0.98.3:44295/ using document root <none> and password file <none>
I20260929 00:59:44.215689 16099 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:44.215772 16099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:44.215976 16099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:44.217950 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "f105184c72654ad8a500398540e02870"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "a82076f85703d9e05f5c2b10370afb37"
server_key_iv: "beebcdbceaac748850b8bb624559538f"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.218541 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f105184c72654ad8a500398540e02870"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "a82076f85703d9e05f5c2b10370afb37"
server_key_iv: "beebcdbceaac748850b8bb624559538f"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.222081 16099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:44.224772 16114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:44.225915 16099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:44.226009 16099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "f105184c72654ad8a500398540e02870"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "a82076f85703d9e05f5c2b10370afb37"
server_key_iv: "beebcdbceaac748850b8bb624559538f"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.226089 16099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:44.256027 16099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:44.259389 16099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:44.259549 16099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:44.260076 16099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:44.260847 16099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:44.260892 16099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:44.260931 16099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:44.260952 16099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:44.269982 16099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42337
I20260929 00:59:44.270005 16227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42337 every 8 connection(s)
I20260929 00:59:44.271013 16099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:44.273376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 16099
I20260929 00:59:44.273502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 00:59:44.273866 392 external_mini_cluster.cc:1428] Setting key 820a5cd27d29f3ca7576013a1d20d11d
Sep 29 00:59:44 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 00:59:44.284320 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.272955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34697 (local address 127.0.98.62:39987)
0929 00:59:44.273092 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.273096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.274313 (+ 1217us) server_negotiation.cc:407] Connection header received
0929 00:59:44.275300 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.275303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.275357 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.275434 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:44.276630 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.277164 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.278328 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.278559 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.278667 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.281030 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:44.281034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:44.281037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:44.281069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:44.282378 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:44.282845 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:44.282850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:44.282852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:44.282934 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:44.283277 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:44.283281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:44.283283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:44.283467 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:44.283570 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.284044 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.284160 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:59:44.285221 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.273338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:34697)
0929 00:59:44.274101 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.274159 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:44.275101 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:44.275601 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:44.275608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:44.276014 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:44.276494 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.276508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.277298 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.277301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.278162 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.278170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.278700 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.278704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.278876 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.279477 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:44.279493 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:44.280873 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:44.282483 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:44.282489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:44.282503 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:44.282743 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:44.283038 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:44.283042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:44.283045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:44.283184 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:44.283567 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:44.283574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:44.283819 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 00:59:44.284005 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 00:59:44.284336 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":145,"threads_started":1}
I20260929 00:59:44.286468 16228 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39987
I20260929 00:59:44.286725 16228 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:44.287200 16228 heartbeater.cc:511] Master 127.0.98.62:39987 requested a full tablet report, sending...
I20260929 00:59:44.288199 15769 ts_manager.cc:194] Registered new tserver with Master: f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
I20260929 00:59:44.288746 15769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34697
I20260929 00:59:44.299228 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:44.308326 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.301119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.0.98.62:39987)
0929 00:59:44.301275 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.301283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.301387 (+ 104us) server_negotiation.cc:407] Connection header received
0929 00:59:44.301477 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.301479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.301517 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.301582 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:44.302305 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.302806 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.303584 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.303790 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.303864 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.305147 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:44.305150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:44.305153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:44.305182 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:44.306690 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:44.307096 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:44.307100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:44.307102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:44.307184 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:44.307443 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:44.307448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:44.307449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:44.307668 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:44.307818 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.308042 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.308155 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:59:44.311141 15769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:59:44.314138 15769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:59:44.328652 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.322937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.0.98.2:33873)
0929 00:59:44.324034 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.324048 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.324073 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:59:44.324147 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.324154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.324370 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.324867 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:44.325028 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.325751 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.327208 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.327973 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.328092 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.328246 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.328400 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.328493 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":123,"threads_started":1}
I20260929 00:59:44.328641 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.322799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.0.1:54794)
0929 00:59:44.323168 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.323226 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:44.323350 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:44.324529 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:44.324532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:44.324551 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:44.324709 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.324720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.325878 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.325881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.327088 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.327098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.328091 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.328095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.328196 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.328230 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:59:44.328370 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 00:59:44.328492 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":65,"thread_start_us":72,"threads_started":1}
I20260929 00:59:44.329177 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.323106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34505 (local address 127.0.0.1:48240)
0929 00:59:44.323724 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.323740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:44.323825 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:44.324508 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:44.324514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:44.324536 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:44.324704 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.324711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.326209 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.326215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.327457 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.327466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.328764 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.328769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.328877 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.328898 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:59:44.328969 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 00:59:44.329038 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":30,"spinlock_wait_cycles":1920,"thread_start_us":91,"threads_started":1}
I20260929 00:59:44.329770 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.323490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42337 (local address 127.0.0.1:60270)
0929 00:59:44.324033 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.324046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:44.324128 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:44.324827 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:44.324830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:44.324846 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:44.324933 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.324939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.326327 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.326330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.327568 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.327576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.328802 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.328805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.329467 (+ 662us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.329485 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 00:59:44.329566 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 00:59:44.329645 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":643,"thread_start_us":89,"threads_started":1}
I20260929 00:59:44.329855 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.323898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.0.98.3:42337)
0929 00:59:44.324292 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.324302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.324319 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:59:44.324385 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.324389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.324544 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.324673 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:44.325067 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.326170 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.327685 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.328467 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.328864 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.329030 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.329601 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.329699 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":130,"threads_started":1}
I20260929 00:59:44.330237 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.323642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.0.98.1:34505)
0929 00:59:44.324048 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.324065 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.324090 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:59:44.324185 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.324192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.324362 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.324520 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:44.324815 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.326017 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.327575 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.328409 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.328533 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.328703 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.329936 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.330048 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":81,"threads_started":1}
I20260929 00:59:44.331929 16028 tablet_service.cc:1467] Processing CreateTablet for tablet 1e26455dc2bf4cb5993233bee9868960 (DEFAULT_TABLE table=test-table [id=88a18e2f28f848fe9ee801b958d940ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:44.332523 15894 tablet_service.cc:1467] Processing CreateTablet for tablet 1e26455dc2bf4cb5993233bee9868960 (DEFAULT_TABLE table=test-table [id=88a18e2f28f848fe9ee801b958d940ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:44.333070 16028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e26455dc2bf4cb5993233bee9868960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:44.332924 16162 tablet_service.cc:1467] Processing CreateTablet for tablet 1e26455dc2bf4cb5993233bee9868960 (DEFAULT_TABLE table=test-table [id=88a18e2f28f848fe9ee801b958d940ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:44.333312 15894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e26455dc2bf4cb5993233bee9868960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:44.333873 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e26455dc2bf4cb5993233bee9868960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:44.339304 16245 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Bootstrap starting.
I20260929 00:59:44.340415 16246 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: Bootstrap starting.
I20260929 00:59:44.340962 16247 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: Bootstrap starting.
I20260929 00:59:44.341656 16245 tablet_bootstrap.cc:654] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:44.343012 16246 tablet_bootstrap.cc:654] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:44.343035 16245 log.cc:824] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:44.343119 16247 tablet_bootstrap.cc:654] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:44.343828 16246 log.cc:824] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:44.343924 16247 log.cc:824] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:44.345492 16246 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: No bootstrap required, opened a new log
I20260929 00:59:44.345510 16245 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: No bootstrap required, opened a new log
I20260929 00:59:44.345633 16247 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: No bootstrap required, opened a new log
I20260929 00:59:44.345700 16245 ts_tablet_manager.cc:1397] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 00:59:44.345701 16246 ts_tablet_manager.cc:1397] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 00:59:44.345772 16247 ts_tablet_manager.cc:1397] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 00:59:44.348560 16246 raft_consensus.cc:348] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.348809 16246 raft_consensus.cc:374] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:44.348881 16246 raft_consensus.cc:729] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba27d79c3b524ac48bd9d176ba54b614, State: Initialized, Role: FOLLOWER
I20260929 00:59:44.349390 16246 consensus_queue.cc:260] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.349502 16247 raft_consensus.cc:348] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.349502 16245 raft_consensus.cc:348] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.349761 16245 raft_consensus.cc:374] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:44.349819 16247 raft_consensus.cc:374] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:44.349854 16245 raft_consensus.cc:729] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca028b96c255482b9b6f3d044a4aa893, State: Initialized, Role: FOLLOWER
I20260929 00:59:44.349911 16247 raft_consensus.cc:729] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f105184c72654ad8a500398540e02870, State: Initialized, Role: FOLLOWER
I20260929 00:59:44.350386 16094 heartbeater.cc:503] Master 127.0.98.62:39987 was elected leader, sending a full tablet report...
I20260929 00:59:44.350378 16245 consensus_queue.cc:260] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.350384 16247 consensus_queue.cc:260] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.350539 16246 ts_tablet_manager.cc:1428] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 00:59:44.351078 15960 heartbeater.cc:503] Master 127.0.98.62:39987 was elected leader, sending a full tablet report...
I20260929 00:59:44.351369 16247 ts_tablet_manager.cc:1428] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:59:44.351369 16245 ts_tablet_manager.cc:1428] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:59:44.351645 16228 heartbeater.cc:503] Master 127.0.98.62:39987 was elected leader, sending a full tablet report...
I20260929 00:59:44.448858 16253 raft_consensus.cc:482] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:44.449195 16253 raft_consensus.cc:504] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.450510 16253 leader_election.cc:288] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), ca028b96c255482b9b6f3d044a4aa893 (127.0.98.1:34505)
I20260929 00:59:44.454284 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.450698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.98.3:50075)
0929 00:59:44.450923 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.450939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:44.451034 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:44.451300 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:44.451303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:44.451320 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:44.451438 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.451443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.452255 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.452261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.453200 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.453209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.453834 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.453837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.453945 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.453963 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 00:59:44.454045 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 00:59:44.454126 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":3}
I20260929 00:59:44.454284 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.450894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50075 (local address 127.0.98.2:33873)
0929 00:59:44.451020 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.451024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.451040 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:59:44.451147 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.451155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.451204 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.451291 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:44.451542 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.452145 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.453315 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.453712 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.453821 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.453882 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.454060 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.454126 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:59:44.454468 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.450884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36781 (local address 127.0.98.1:34505)
0929 00:59:44.451021 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.451026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.451260 (+ 234us) server_negotiation.cc:407] Connection header received
0929 00:59:44.451410 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.451413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.451485 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.451573 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:44.451872 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.452405 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.453512 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.453905 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.454017 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.454080 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.454286 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.454366 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:59:44.454475 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.450799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34505 (local address 127.0.98.3:36781)
0929 00:59:44.451166 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.451177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:44.451298 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:44.451581 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:44.451584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:44.451599 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:44.451751 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.451757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.452514 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.452521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.453394 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:44.453403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.454018 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.454022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:44.454124 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.454149 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:59:44.454260 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:59:44.454330 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260929 00:59:44.455192 15914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e26455dc2bf4cb5993233bee9868960" candidate_uuid: "f105184c72654ad8a500398540e02870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca028b96c255482b9b6f3d044a4aa893" is_pre_election: true
I20260929 00:59:44.455192 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e26455dc2bf4cb5993233bee9868960" candidate_uuid: "f105184c72654ad8a500398540e02870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba27d79c3b524ac48bd9d176ba54b614" is_pre_election: true
I20260929 00:59:44.455519 16048 raft_consensus.cc:2454] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f105184c72654ad8a500398540e02870 in term 0.
I20260929 00:59:44.455519 15914 raft_consensus.cc:2454] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f105184c72654ad8a500398540e02870 in term 0.
I20260929 00:59:44.456040 16118 leader_election.cc:302] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ba27d79c3b524ac48bd9d176ba54b614, ca028b96c255482b9b6f3d044a4aa893, f105184c72654ad8a500398540e02870; no voters:
I20260929 00:59:44.456379 16253 raft_consensus.cc:2781] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:44.456458 16253 raft_consensus.cc:482] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:44.456488 16253 raft_consensus.cc:3037] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:44.457556 16253 raft_consensus.cc:504] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.457926 16253 leader_election.cc:288] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [CANDIDATE]: Term 1 election: Requested vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), ca028b96c255482b9b6f3d044a4aa893 (127.0.98.1:34505)
I20260929 00:59:44.458333 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e26455dc2bf4cb5993233bee9868960" candidate_uuid: "f105184c72654ad8a500398540e02870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba27d79c3b524ac48bd9d176ba54b614"
I20260929 00:59:44.458346 15914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e26455dc2bf4cb5993233bee9868960" candidate_uuid: "f105184c72654ad8a500398540e02870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca028b96c255482b9b6f3d044a4aa893"
I20260929 00:59:44.458438 16048 raft_consensus.cc:3037] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:44.458493 15914 raft_consensus.cc:3037] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:44.459476 16048 raft_consensus.cc:2454] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f105184c72654ad8a500398540e02870 in term 1.
I20260929 00:59:44.459507 15914 raft_consensus.cc:2454] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f105184c72654ad8a500398540e02870 in term 1.
I20260929 00:59:44.459816 16117 leader_election.cc:302] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba27d79c3b524ac48bd9d176ba54b614, f105184c72654ad8a500398540e02870; no voters:
I20260929 00:59:44.460055 16253 raft_consensus.cc:2781] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:44.460285 16253 raft_consensus.cc:686] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 1 LEADER]: Becoming Leader. State: Replica: f105184c72654ad8a500398540e02870, State: Running, Role: LEADER
I20260929 00:59:44.460603 16253 consensus_queue.cc:237] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 00:59:44.463816 15768 catalog_manager.cc:5495] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 reported cstate change: term changed from 0 to 1, leader changed from <none> to f105184c72654ad8a500398540e02870 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f105184c72654ad8a500398540e02870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } health_report { overall_health: UNKNOWN } } }
I20260929 00:59:44.485036 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.481351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.0.98.3:42337)
0929 00:59:44.481480 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.481487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.481503 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:59:44.481684 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.481686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.481744 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.481877 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:44.482145 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.482711 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.483615 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.483837 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.483943 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.484237 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:44.484645 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:44.484777 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.484815 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.484869 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:59:44.490674 16048 raft_consensus.cc:1261] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 1 FOLLOWER]: Refusing update from remote peer f105184c72654ad8a500398540e02870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:44.490675 15914 raft_consensus.cc:1261] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Refusing update from remote peer f105184c72654ad8a500398540e02870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:44.491406 16253 consensus_queue.cc:1035] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:44.491597 16255 consensus_queue.cc:1035] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:44.499879 16259 mvcc.cc:205] Tried to move back new op lower bound from 7334476122066690048 to 7334476121952862208. Current Snapshot: MvccSnapshot[applied={T|T < 7334476122066690048}]
I20260929 00:59:44.500370 16258 mvcc.cc:205] Tried to move back new op lower bound from 7334476122066690048 to 7334476121952862208. Current Snapshot: MvccSnapshot[applied={T|T < 7334476122066690048}]
I20260929 00:59:44.508128 15768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:44.508311 15768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:44.510744 15768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Sending DeleteTablet for 3 replicas of tablet 1e26455dc2bf4cb5993233bee9868960
I20260929 00:59:44.511584 16028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e26455dc2bf4cb5993233bee9868960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54794
I20260929 00:59:44.511577 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e26455dc2bf4cb5993233bee9868960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60270
I20260929 00:59:44.512071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 15831
I20260929 00:59:44.512054 15894 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e26455dc2bf4cb5993233bee9868960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48240
I20260929 00:59:44.512964 16269 tablet_replica.cc:330] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: stopping tablet replica
I20260929 00:59:44.513294 16269 raft_consensus.cc:2229] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:59:44.513705 16269 raft_consensus.cc:2258] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:44.514077 16268 tablet_replica.cc:330] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: stopping tablet replica
I20260929 00:59:44.514457 16268 raft_consensus.cc:2229] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:44.514809 16268 raft_consensus.cc:2258] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:44.515298 16269 ts_tablet_manager.cc:1905] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:44.516476 16268 ts_tablet_manager.cc:1905] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:44.518106 16269 ts_tablet_manager.cc:1918] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:44.518301 16269 log.cc:1196] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/1e26455dc2bf4cb5993233bee9868960
I20260929 00:59:44.518728 16269 ts_tablet_manager.cc:1939] T 1e26455dc2bf4cb5993233bee9868960 P f105184c72654ad8a500398540e02870: Deleting consensus metadata
I20260929 00:59:44.520000 15755 catalog_manager.cc:4841] TS f105184c72654ad8a500398540e02870 (127.0.98.3:42337): tablet 1e26455dc2bf4cb5993233bee9868960 (table test-table [id=88a18e2f28f848fe9ee801b958d940ca]) successfully deleted
I20260929 00:59:44.520179 16268 ts_tablet_manager.cc:1918] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:44.520295 16268 log.cc:1196] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/1e26455dc2bf4cb5993233bee9868960
I20260929 00:59:44.520689 16268 ts_tablet_manager.cc:1939] T 1e26455dc2bf4cb5993233bee9868960 P ba27d79c3b524ac48bd9d176ba54b614: Deleting consensus metadata
W20260929 00:59:44.521198 15756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34505 (error 108)
I20260929 00:59:44.521440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 15965
I20260929 00:59:44.523697 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.523409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34505 (local address 127.0.0.1:48242)
0929 00:59:44.523542 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.523620 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:59:44.523895 15756 catalog_manager.cc:4584] TS ca028b96c255482b9b6f3d044a4aa893 (127.0.98.1:34505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e26455dc2bf4cb5993233bee9868960: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34505: connect: Connection refused (error 111)
I20260929 00:59:44.528544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 16099
I20260929 00:59:44.528755 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.528493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.0.1:54798)
0929 00:59:44.528627 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:44.528678 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:59:44.534085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 15737
I20260929 01:00:04.540719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39987
--webserver_interface=127.0.98.62
--webserver_port=37341
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:04.650619 16271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:04.650919 16271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:04.651006 16271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:04.655202 16271 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:00:04.655282 16271 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:00:04.655329 16271 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:00:04.655372 16271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:04.655433 16271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:04.655474 16271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:04.655508 16271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:04.655550 16271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:04.660962 16271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:04.662082 16271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:04.663172 16271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:04.676263 16278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:04.676254 16277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:04.676338 16280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:04.676435 16271 server_base.cc:1034] running on GCE node
I20260929 01:00:04.677222 16271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:04.678423 16271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:04.679612 16271 hybrid_clock.cc:648] HybridClock initialized: now 1790643604679576 us; error 48 us; skew 500 ppm
Sep 29 01:00:04 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:04.686527 16271 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:00:04.687496 16271 webserver.cc:460] Webserver started at http://127.0.98.62:37341/ using document root <none> and password file <none>
I20260929 01:00:04.688042 16271 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:04.688110 16271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:04.691952 16271 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 01:00:04.694069 16287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:04.695392 16271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:04.695515 16271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "45f3d8faa5614970bbb3d8c78f009511"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "8c0333661b6ba850b7c02dec58e4bd17"
server_key_iv: "6372fba098c307bb944630779c508e67"
server_key_version: "encryptionkey@0"
I20260929 01:00:04.696108 16271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:04.706596 16271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:04.709964 16271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:04.710177 16271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:04.717734 16271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39987
I20260929 01:00:04.718024 16339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39987 every 8 connection(s)
I20260929 01:00:04.718904 16271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:04.723217 16340 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:00:04.725100 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Bootstrap starting.
I20260929 01:00:04.727370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 16271
I20260929 01:00:04.727803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34505
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39987
--webserver_port=33347
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:00:04.743366 16340 log.cc:824] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:04.752947 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:00:04.753384 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Bootstrap complete.
I20260929 01:00:04.757925 16340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 01:00:04.758455 16340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f3d8faa5614970bbb3d8c78f009511, State: Initialized, Role: FOLLOWER
I20260929 01:00:04.758983 16340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 01:00:04.759138 16340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:04.759243 16340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:04.759373 16340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:00:04.761446 16340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 01:00:04.761834 16340 leader_election.cc:302] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f3d8faa5614970bbb3d8c78f009511; no voters:
I20260929 01:00:04.762220 16340 leader_election.cc:288] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:00:04.762331 16344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:00:04.762598 16344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 2 LEADER]: Becoming Leader. State: Replica: 45f3d8faa5614970bbb3d8c78f009511, State: Running, Role: LEADER
I20260929 01:00:04.762915 16344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 01:00:04.763550 16340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:04.764495 16346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f3d8faa5614970bbb3d8c78f009511. Latest consensus state: current_term: 2 leader_uuid: "45f3d8faa5614970bbb3d8c78f009511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } } }
I20260929 01:00:04.764526 16345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "45f3d8faa5614970bbb3d8c78f009511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } } }
I20260929 01:00:04.764639 16345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:04.764642 16346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:04.769169 16359 catalog_manager.cc:1259] Loaded cluster ID: dede7f9ec350415abcd2021afdcea247
I20260929 01:00:04.769232 16359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: loading cluster ID for follower catalog manager: success
I20260929 01:00:04.771763 16359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: acquiring CA information for follower catalog manager: success
I20260929 01:00:04.773313 16359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:00:04.773509 16352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:04.776575 16352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:04.777060 16352 catalog_manager.cc:1259] Loaded cluster ID: dede7f9ec350415abcd2021afdcea247
I20260929 01:00:04.777124 16352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:04.777817 16352 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:04.778321 16352 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Loaded TSK: 0
I20260929 01:00:04.793177 16352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Generated new TSK 1
I20260929 01:00:04.793468 16352 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:00:04.851143 16342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:04.851462 16342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:04.851567 16342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:04.855732 16342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:04.855820 16342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:04.855933 16342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:00:04.861255 16342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33347
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:04.862414 16342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:04.863484 16342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:04.875036 16368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:04.875092 16367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:04.875350 16342 server_base.cc:1034] running on GCE node
W20260929 01:00:04.875406 16370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:04.875778 16342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:04.876513 16342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:04.877765 16342 hybrid_clock.cc:648] HybridClock initialized: now 1790643604877742 us; error 54 us; skew 500 ppm
Sep 29 01:00:04 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:04.880630 16342 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:00:04.881669 16342 webserver.cc:460] Webserver started at http://127.0.98.1:33347/ using document root <none> and password file <none>
I20260929 01:00:04.882298 16342 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:04.882376 16342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:04.886351 16342 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:00:04.888595 16377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:04.889918 16342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:04.890023 16342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "ca028b96c255482b9b6f3d044a4aa893"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "5217746c931a990d52cb3277cf6c77ae"
server_key_iv: "742ad1b84cf3de61c5144d68bb452f94"
server_key_version: "encryptionkey@0"
I20260929 01:00:04.890694 16342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:04.916574 16342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:04.920053 16342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:04.920269 16342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:04.920956 16342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:04.922128 16384 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 01:00:04.926136 16342 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 01:00:04.926247 16342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260929 01:00:04.926335 16342 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 01:00:04.928051 16342 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 01:00:04.928107 16342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 01:00:04.928361 16384 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Bootstrap starting.
I20260929 01:00:04.939144 16342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34505
I20260929 01:00:04.939358 16491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34505 every 8 connection(s)
I20260929 01:00:04.940424 16342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:04.949533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 16342
I20260929 01:00:04.949744 16384 log.cc:824] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:04.950143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33873
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39987
--webserver_port=34403
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:00:04 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:04.959458 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:04.942565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:49675)
0929 01:00:04.943496 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:04.943555 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:04.944729 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:04.945927 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:04.945939 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:04.946539 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:04.947163 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:04.947185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:04.948519 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:04.948522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:04.949822 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:04.949837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:04.951081 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:04.951089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:04.951367 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:04.952161 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:04.952179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:04.953711 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:04.956096 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:04.956102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:04.956115 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:04.956403 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:04.956797 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:04.956802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:04.956804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:04.956954 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:04.957575 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:04.957581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:04.957862 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 01:00:04.958061 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 01:00:04.958452 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":113,"threads_started":1}
I20260929 01:00:04.959445 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:04.942563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49675 (local address 127.0.98.62:39987)
0929 01:00:04.943111 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:04.943133 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:04.943741 (+ 608us) server_negotiation.cc:407] Connection header received
0929 01:00:04.945167 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:04.945178 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:04.945484 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:04.945784 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:04.947391 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:04.948373 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:04.949980 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:04.950885 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:04.951883 (+ 998us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:04.953940 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:04.953946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:04.953959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:04.953990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:04.955944 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:04.956551 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:04.956556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:04.956560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:04.956685 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:04.957062 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:04.957067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:04.957069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:04.957446 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:04.957604 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:04.958224 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 01:00:04.958529 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":125,"threads_started":1}
I20260929 01:00:04.959769 16384 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:00:04.960256 16384 tablet_bootstrap.cc:492] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Bootstrap complete.
I20260929 01:00:04.961783 16384 ts_tablet_manager.cc:1397] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.005s
I20260929 01:00:04.962677 16492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39987
I20260929 01:00:04.962950 16492 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:04.963557 16492 heartbeater.cc:511] Master 127.0.98.62:39987 requested a full tablet report, sending...
I20260929 01:00:04.964998 16384 raft_consensus.cc:348] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:04.965521 16384 raft_consensus.cc:729] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca028b96c255482b9b6f3d044a4aa893, State: Initialized, Role: FOLLOWER
I20260929 01:00:04.966070 16384 consensus_queue.cc:260] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:04.966522 16304 ts_manager.cc:194] Registered new tserver with Master: ca028b96c255482b9b6f3d044a4aa893 (127.0.98.1:34505)
I20260929 01:00:04.967064 16384 ts_tablet_manager.cc:1428] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Time spent starting tablet: real 0.005s user 0.004s sys 0.004s
I20260929 01:00:04.970149 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49675
I20260929 01:00:04.973206 16492 heartbeater.cc:503] Master 127.0.98.62:39987 was elected leader, sending a full tablet report...
Sep 29 01:00:04 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:00:04.981375 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:04.969487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.0.98.1:34505)
0929 01:00:04.969850 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:04.969865 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:04.970412 (+ 547us) server_negotiation.cc:407] Connection header received
0929 01:00:04.970632 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:04.970638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:04.970859 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:04.971015 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:04.971780 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:04.972984 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:04.974287 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:04.974856 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:04.974971 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:04.977488 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:04.977498 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:04.977508 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:04.977533 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:04.979360 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:04.979918 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:04.979923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:04.979924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:04.980008 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:04.980338 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:04.980341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:04.980342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:04.980579 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:04.980680 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:04.981088 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 01:00:04.981240 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":275,"thread_start_us":102,"threads_started":1}
I20260929 01:00:04.981382 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:04.969392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34505 (local address 127.0.0.1:53376)
0929 01:00:04.970273 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:04.970325 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:04.970470 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:04.971027 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:04.971033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:04.971499 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:04.971645 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:04.971653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:04.973121 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:04.973124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:04.974128 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:04.974139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:04.974955 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:04.974972 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:04.975099 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:04.975760 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:04.975774 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:04.977338 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:04.979484 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:04.979493 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:04.979499 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:04.979797 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:04.980103 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:04.980106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:04.980139 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:04.980240 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:04.980692 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:04.980698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:04.980833 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 01:00:04.980992 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 01:00:04.981206 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":111,"threads_started":1}
I20260929 01:00:04.982558 16426 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e26455dc2bf4cb5993233bee9868960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53376
I20260929 01:00:04.983469 16501 tablet_replica.cc:330] stopping tablet replica
I20260929 01:00:04.983793 16501 raft_consensus.cc:2229] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:04.984066 16501 raft_consensus.cc:2258] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:04.985173 16501 ts_tablet_manager.cc:1905] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:04.989946 16501 ts_tablet_manager.cc:1918] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:04.990064 16501 log.cc:1196] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal/wals/1e26455dc2bf4cb5993233bee9868960
I20260929 01:00:04.990353 16501 ts_tablet_manager.cc:1939] T 1e26455dc2bf4cb5993233bee9868960 P ca028b96c255482b9b6f3d044a4aa893: Deleting consensus metadata
I20260929 01:00:04.992619 16291 catalog_manager.cc:4841] TS ca028b96c255482b9b6f3d044a4aa893 (127.0.98.1:34505): tablet 1e26455dc2bf4cb5993233bee9868960 (table test-table [id=88a18e2f28f848fe9ee801b958d940ca]) successfully deleted
W20260929 01:00:05.085570 16497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:05.086004 16497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:05.086099 16497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:05.094910 16497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:05.095018 16497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:05.095157 16497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:00:05.105072 16497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34403
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:05.106251 16497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:05.107398 16497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:05.117833 16506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:05.117902 16507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:05.117942 16509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:05.117863 16497 server_base.cc:1034] running on GCE node
I20260929 01:00:05.118594 16497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:05.119328 16497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:05.120554 16497 hybrid_clock.cc:648] HybridClock initialized: now 1790643605120538 us; error 30 us; skew 500 ppm
Sep 29 01:00:05 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:05.123757 16497 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:00:05.124718 16497 webserver.cc:460] Webserver started at http://127.0.98.2:34403/ using document root <none> and password file <none>
I20260929 01:00:05.125341 16497 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:05.125458 16497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:05.129473 16497 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 01:00:05.131860 16516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:05.133104 16497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 01:00:05.133242 16497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "ba27d79c3b524ac48bd9d176ba54b614"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "29c2bf04f011bca2c94697ec2cd1148a"
server_key_iv: "03d8281b4222a2b40bc69f04758bec52"
server_key_version: "encryptionkey@0"
I20260929 01:00:05.133837 16497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:05.161202 16497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:05.164927 16497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:05.165153 16497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:05.165894 16497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:05.166827 16497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:05.166917 16497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:05.166994 16497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:05.167028 16497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:05.177163 16497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33873
I20260929 01:00:05.177182 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33873 every 8 connection(s)
I20260929 01:00:05.178364 16497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:05.188675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 16497
I20260929 01:00:05.189270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42337
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39987
--webserver_port=44295
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:00:05 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:05.195221 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.180394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47003 (local address 127.0.98.62:39987)
0929 01:00:05.180574 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.180578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.181690 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 01:00:05.182681 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.182684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.182735 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.182818 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:05.184086 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.184698 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.186268 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.186663 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.186811 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.190222 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:05.190229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:05.190236 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:05.190281 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:05.192401 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:05.193171 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:05.193173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:05.193175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:05.193266 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:05.193738 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:05.193743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:05.193749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:05.194004 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:05.194095 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.194904 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.195063 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:00:05.196612 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.180825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:47003)
0929 01:00:05.181521 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:05.181566 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:05.182481 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:05.183026 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:05.183033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:05.183425 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:05.183927 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.183941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.184838 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.184841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.186062 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.186076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.186878 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.186883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.187126 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.187901 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:05.187924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:05.189991 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:05.192572 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:05.192581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:05.192604 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:05.193052 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:05.193380 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:05.193386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:05.193391 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:05.193632 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:05.194127 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:05.194137 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:05.194589 (+ 452us) client_negotiation.cc:769] Sending connection context
0929 01:00:05.194841 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 01:00:05.195305 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":108,"threads_started":1}
I20260929 01:00:05.197994 16630 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39987
I20260929 01:00:05.198350 16630 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:05.198909 16630 heartbeater.cc:511] Master 127.0.98.62:39987 requested a full tablet report, sending...
I20260929 01:00:05.200215 16304 ts_manager.cc:194] Registered new tserver with Master: ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873)
I20260929 01:00:05.200927 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47003
W20260929 01:00:05.303380 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:05.303727 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:05.303825 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:05.307992 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:05.308092 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:05.308184 16633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:00:05.313727 16633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44295
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:05.314954 16633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:05.316047 16633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:05.326275 16641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:05.326267 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:05.326326 16638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:05.326474 16633 server_base.cc:1034] running on GCE node
I20260929 01:00:05.326980 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:05.327677 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:05.328879 16633 hybrid_clock.cc:648] HybridClock initialized: now 1790643605328864 us; error 30 us; skew 500 ppm
Sep 29 01:00:05 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:05.331619 16633 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:00:05.332464 16633 webserver.cc:460] Webserver started at http://127.0.98.3:44295/ using document root <none> and password file <none>
I20260929 01:00:05.333012 16633 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:05.333122 16633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:05.336936 16633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:00:05.339037 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:05.340360 16633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:05.340507 16633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "f105184c72654ad8a500398540e02870"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "a82076f85703d9e05f5c2b10370afb37"
server_key_iv: "beebcdbceaac748850b8bb624559538f"
server_key_version: "encryptionkey@0"
I20260929 01:00:05.341084 16633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:05.364267 16633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:05.367494 16633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:05.367712 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:05.368345 16633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:05.369197 16633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:05.369285 16633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:05.369357 16633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:05.369410 16633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:05.378166 16633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42337
I20260929 01:00:05.378214 16761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42337 every 8 connection(s)
I20260929 01:00:05.379268 16633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:05.387523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 16633
Sep 29 01:00:05 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:05.393647 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.381128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36479 (local address 127.0.98.62:39987)
0929 01:00:05.381269 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.381273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.382361 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 01:00:05.383255 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.383257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.383315 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.383391 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:05.384616 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.385205 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.386365 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.386612 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.386702 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.389714 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:05.389717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:05.389719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:05.389741 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:05.391417 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:05.391990 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:05.391994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:05.391996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:05.392090 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:05.392498 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:05.392501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:05.392502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:05.392670 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:05.392784 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.393337 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.393461 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:00:05.394773 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.381514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:36479)
0929 01:00:05.382191 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:05.382265 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:05.383051 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:05.383570 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:05.383577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:05.383939 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:05.384431 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.384444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.385336 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.385339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.386244 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.386252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.386746 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.386749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.386885 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.387516 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:05.387538 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:05.389530 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:05.391543 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:05.391550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:05.391567 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:05.391889 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:05.392217 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:05.392221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:05.392223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:05.392399 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:05.392785 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:05.392795 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:05.393074 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 01:00:05.393272 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 01:00:05.393604 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":122,"threads_started":1}
I20260929 01:00:05.396052 16762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39987
I20260929 01:00:05.396312 16762 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:05.396908 16762 heartbeater.cc:511] Master 127.0.98.62:39987 requested a full tablet report, sending...
I20260929 01:00:05.397004 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.388222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.0.98.62:39987)
0929 01:00:05.388550 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.388553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.388632 (+ 79us) server_negotiation.cc:407] Connection header received
0929 01:00:05.388810 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.388813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.388856 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.388940 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:05.389843 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.390749 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.391794 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.392138 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.392239 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.393597 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:05.393600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:05.393604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:05.393632 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:05.395330 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:05.395746 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:05.395751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:05.395753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:05.395835 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:05.396166 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:05.396172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:05.396176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:05.396379 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:05.396568 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.396757 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.396855 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":84,"threads_started":1}
I20260929 01:00:05.398063 16304 ts_manager.cc:194] Registered new tserver with Master: f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
I20260929 01:00:05.398772 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36479
I20260929 01:00:05.398927 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:05.403298 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.399567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49576 (local address 127.0.98.62:39987)
0929 01:00:05.399687 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.399691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.399945 (+ 254us) server_negotiation.cc:407] Connection header received
0929 01:00:05.400133 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.400136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.400234 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.400335 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:05.400585 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.401197 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.402066 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.402382 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.402502 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.402792 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:05.402920 (+ 128us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 01:00:05.403177 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49576: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 01:00:05.403365 16765 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49576: token expired
W20260929 01:00:05.403723 16232 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39987)
I20260929 01:00:05.411134 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.404146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.0.98.62:39987)
0929 01:00:05.404252 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.404255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.404292 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:00:05.404396 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.404398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.404431 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.404494 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:05.405093 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.405576 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.406385 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.406586 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.406664 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.407919 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:05.407922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:05.407923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:05.407944 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:05.409444 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:05.409834 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:05.409837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:05.409838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:05.409955 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:05.410280 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:05.410283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:05.410284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:05.410461 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:05.410574 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.410819 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.410978 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:00:05.413769 16304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:05.417141 16304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:05.425649 16426 tablet_service.cc:1467] Processing CreateTablet for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89 (DEFAULT_TABLE table=test-table [id=f261182ee6a9443ab4a5258b9cc2ed82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:05.426368 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:05.429207 16384 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Bootstrap starting.
I20260929 01:00:05.430030 16384 tablet_bootstrap.cc:654] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:05.430285 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.424345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.0.1:35724)
0929 01:00:05.424477 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:05.424495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:05.424644 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:05.425991 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:05.425994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:05.426011 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:05.426128 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.426138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.427433 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.427437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.428445 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.428455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.429752 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.429756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.429849 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.429908 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 01:00:05.430011 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:00:05.430108 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":34}
I20260929 01:00:05.430882 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.424543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.0.98.2:33873)
0929 01:00:05.425479 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.425492 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.425515 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:00:05.425587 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.425593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.425795 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.426005 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:05.426360 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.427301 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.428588 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.429605 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.429734 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.429910 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.430616 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.430733 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":188,"threads_started":1}
I20260929 01:00:05.431195 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.424613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42337 (local address 127.0.0.1:40934)
0929 01:00:05.425000 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:05.425016 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:05.425112 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:05.425711 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:05.425714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:05.425734 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:05.426089 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.426096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.427499 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.427502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.428618 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.428629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.430731 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.430735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.430830 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.430851 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:00:05.430952 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 01:00:05.431038 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":1,"thread_start_us":143,"threads_started":1}
I20260929 01:00:05.431515 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.424807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.0.98.3:42337)
0929 01:00:05.425140 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.425156 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.425186 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:00:05.425326 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.425332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.425548 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.425734 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:05.426300 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.427348 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.428771 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.429721 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.429987 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.430272 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.431275 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.431371 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":127,"threads_started":1}
I20260929 01:00:05.432109 16384 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: No bootstrap required, opened a new log
I20260929 01:00:05.432229 16384 ts_tablet_manager.cc:1397] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 01:00:05.432672 16384 raft_consensus.cc:348] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.432807 16384 raft_consensus.cc:374] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:05.432861 16384 raft_consensus.cc:729] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca028b96c255482b9b6f3d044a4aa893, State: Initialized, Role: FOLLOWER
I20260929 01:00:05.433008 16384 consensus_queue.cc:260] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.433348 16384 ts_tablet_manager.cc:1428] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 01:00:05.434051 16696 tablet_service.cc:1467] Processing CreateTablet for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89 (DEFAULT_TABLE table=test-table [id=f261182ee6a9443ab4a5258b9cc2ed82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:05.434054 16564 tablet_service.cc:1467] Processing CreateTablet for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89 (DEFAULT_TABLE table=test-table [id=f261182ee6a9443ab4a5258b9cc2ed82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:05.435195 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:05.435189 16564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 01:00:05.441684 16493 tablet.cc:2369] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:05.443272 16776 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Bootstrap starting.
I20260929 01:00:05.443749 16775 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Bootstrap starting.
I20260929 01:00:05.445948 16776 tablet_bootstrap.cc:654] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:05.446485 16775 tablet_bootstrap.cc:654] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:05.447329 16775 log.cc:824] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:05.447329 16776 log.cc:824] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:05.449323 16775 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: No bootstrap required, opened a new log
I20260929 01:00:05.449334 16776 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: No bootstrap required, opened a new log
I20260929 01:00:05.449540 16775 ts_tablet_manager.cc:1397] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:05.449559 16776 ts_tablet_manager.cc:1397] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:05.452411 16776 raft_consensus.cc:348] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.452448 16775 raft_consensus.cc:348] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.452647 16776 raft_consensus.cc:374] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:05.452680 16775 raft_consensus.cc:374] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:05.452708 16776 raft_consensus.cc:729] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f105184c72654ad8a500398540e02870, State: Initialized, Role: FOLLOWER
I20260929 01:00:05.452762 16775 raft_consensus.cc:729] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba27d79c3b524ac48bd9d176ba54b614, State: Initialized, Role: FOLLOWER
I20260929 01:00:05.453181 16775 consensus_queue.cc:260] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.453181 16776 consensus_queue.cc:260] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.453819 16630 heartbeater.cc:503] Master 127.0.98.62:39987 was elected leader, sending a full tablet report...
I20260929 01:00:05.453824 16762 heartbeater.cc:503] Master 127.0.98.62:39987 was elected leader, sending a full tablet report...
I20260929 01:00:05.454183 16775 ts_tablet_manager.cc:1428] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:00:05.455302 16776 ts_tablet_manager.cc:1428] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 01:00:05.630297 16763 tablet.cc:2369] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:05.679632 16631 tablet.cc:2369] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:05.700208 16498 raft_consensus.cc:482] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:05.700538 16498 raft_consensus.cc:504] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.701761 16498 leader_election.cc:288] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
I20260929 01:00:05.705904 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.702115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37983 (local address 127.0.98.2:33873)
0929 01:00:05.702314 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.702319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.702484 (+ 165us) server_negotiation.cc:407] Connection header received
0929 01:00:05.702701 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.702705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.702765 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.702817 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:05.703279 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.703781 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.704891 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.705278 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.705399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.705461 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.705690 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.705769 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 01:00:05.705905 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.702604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46595 (local address 127.0.98.3:42337)
0929 01:00:05.702729 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.702733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.702819 (+ 86us) server_negotiation.cc:407] Connection header received
0929 01:00:05.702930 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.702932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.702981 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.703056 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:05.703305 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.703796 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.704892 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.705311 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.705431 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.705488 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.705700 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.705781 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:00:05.705956 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.702478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42337 (local address 127.0.98.1:46595)
0929 01:00:05.702744 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:05.702756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:05.702848 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:05.703074 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:05.703076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:05.703089 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:05.703177 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.703181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.703918 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.703924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.704755 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.704763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.705448 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.705451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.705563 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.705591 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:00:05.705671 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:00:05.705798 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":36,"thread_start_us":81,"threads_started":1}
I20260929 01:00:05.705979 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.702035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.98.1:37983)
0929 01:00:05.702339 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:05.702354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:05.702530 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:05.702968 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:05.702972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:05.702991 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:05.703151 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.703158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.703895 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.703901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.704762 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:05.704771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.705403 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.705407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:05.705504 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.705583 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 01:00:05.705675 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 01:00:05.705808 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":123,"thread_start_us":94,"threads_started":1}
I20260929 01:00:05.706614 16584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c1d24e7bb7406ba6a95cbacc1b1e89" candidate_uuid: "ca028b96c255482b9b6f3d044a4aa893" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba27d79c3b524ac48bd9d176ba54b614" is_pre_election: true
I20260929 01:00:05.706655 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c1d24e7bb7406ba6a95cbacc1b1e89" candidate_uuid: "ca028b96c255482b9b6f3d044a4aa893" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f105184c72654ad8a500398540e02870" is_pre_election: true
I20260929 01:00:05.706925 16584 raft_consensus.cc:2454] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca028b96c255482b9b6f3d044a4aa893 in term 0.
I20260929 01:00:05.706931 16716 raft_consensus.cc:2454] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca028b96c255482b9b6f3d044a4aa893 in term 0.
I20260929 01:00:05.707396 16380 leader_election.cc:302] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba27d79c3b524ac48bd9d176ba54b614, ca028b96c255482b9b6f3d044a4aa893; no voters:
I20260929 01:00:05.707684 16498 raft_consensus.cc:2781] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:05.707762 16498 raft_consensus.cc:482] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:05.707792 16498 raft_consensus.cc:3037] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:05.708809 16498 raft_consensus.cc:504] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.709143 16498 leader_election.cc:288] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 1 election: Requested vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
I20260929 01:00:05.709520 16584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c1d24e7bb7406ba6a95cbacc1b1e89" candidate_uuid: "ca028b96c255482b9b6f3d044a4aa893" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba27d79c3b524ac48bd9d176ba54b614"
I20260929 01:00:05.709633 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c1d24e7bb7406ba6a95cbacc1b1e89" candidate_uuid: "ca028b96c255482b9b6f3d044a4aa893" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f105184c72654ad8a500398540e02870"
I20260929 01:00:05.709676 16584 raft_consensus.cc:3037] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:05.709777 16716 raft_consensus.cc:3037] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:05.710947 16584 raft_consensus.cc:2454] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca028b96c255482b9b6f3d044a4aa893 in term 1.
I20260929 01:00:05.711002 16716 raft_consensus.cc:2454] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca028b96c255482b9b6f3d044a4aa893 in term 1.
I20260929 01:00:05.711275 16380 leader_election.cc:302] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba27d79c3b524ac48bd9d176ba54b614, ca028b96c255482b9b6f3d044a4aa893; no voters:
I20260929 01:00:05.711509 16498 raft_consensus.cc:2781] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:05.711742 16498 raft_consensus.cc:686] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 LEADER]: Becoming Leader. State: Replica: ca028b96c255482b9b6f3d044a4aa893, State: Running, Role: LEADER
I20260929 01:00:05.712007 16498 consensus_queue.cc:237] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:05.715109 16302 catalog_manager.cc:5495] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca028b96c255482b9b6f3d044a4aa893 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ca028b96c255482b9b6f3d044a4aa893" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } health_report { overall_health: HEALTHY } } }
I20260929 01:00:05.795684 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.791769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.0.98.1:34505)
0929 01:00:05.792076 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:05.792083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:05.792096 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:00:05.792179 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:05.792181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:05.792235 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:05.792348 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:05.792636 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.793134 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.794026 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:05.794266 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:05.794347 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:05.794682 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:05.795208 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:05.795300 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:05.795422 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 01:00:05.795473 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":130,"threads_started":1}
I20260929 01:00:05.800009 16584 raft_consensus.cc:1261] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 1 FOLLOWER]: Refusing update from remote peer ca028b96c255482b9b6f3d044a4aa893: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:05.800009 16716 raft_consensus.cc:1261] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 1 FOLLOWER]: Refusing update from remote peer ca028b96c255482b9b6f3d044a4aa893: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:05.800854 16783 consensus_queue.cc:1035] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:05.801048 16498 consensus_queue.cc:1035] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:05.808197 16787 mvcc.cc:205] Tried to move back new op lower bound from 7334476209348689920 to 7334476208998694912. Current Snapshot: MvccSnapshot[applied={T|T < 7334476209348689920}]
I20260929 01:00:05.809409 16788 mvcc.cc:205] Tried to move back new op lower bound from 7334476209348689920 to 7334476208998694912. Current Snapshot: MvccSnapshot[applied={T|T < 7334476209348689920}]
I20260929 01:00:05.809782 16789 mvcc.cc:205] Tried to move back new op lower bound from 7334476209348689920 to 7334476208998694912. Current Snapshot: MvccSnapshot[applied={T|T < 7334476209348689920}]
I20260929 01:00:05.817507 16304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:05.817698 16304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:05.820204 16304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Sending DeleteTablet for 3 replicas of tablet 63c1d24e7bb7406ba6a95cbacc1b1e89
I20260929 01:00:05.821017 16696 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40934
I20260929 01:00:05.821009 16564 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35724
I20260929 01:00:05.821130 16426 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53376
I20260929 01:00:05.821570 16798 tablet_replica.cc:330] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: stopping tablet replica
I20260929 01:00:05.821593 16797 tablet_replica.cc:330] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: stopping tablet replica
I20260929 01:00:05.821712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 16342
I20260929 01:00:05.821909 16797 raft_consensus.cc:2229] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:05.821925 16798 raft_consensus.cc:2229] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:05.822265 16797 raft_consensus.cc:2258] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:05.822268 16798 raft_consensus.cc:2258] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:05.823792 16798 ts_tablet_manager.cc:1905] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:05.823822 16797 ts_tablet_manager.cc:1905] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:05.826030 16797 ts_tablet_manager.cc:1918] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:05.826144 16797 log.cc:1196] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/63c1d24e7bb7406ba6a95cbacc1b1e89
I20260929 01:00:05.826481 16797 ts_tablet_manager.cc:1939] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P f105184c72654ad8a500398540e02870: Deleting consensus metadata
I20260929 01:00:05.827255 16798 ts_tablet_manager.cc:1918] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:05.827358 16798 log.cc:1196] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/63c1d24e7bb7406ba6a95cbacc1b1e89
I20260929 01:00:05.827687 16798 ts_tablet_manager.cc:1939] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ba27d79c3b524ac48bd9d176ba54b614: Deleting consensus metadata
I20260929 01:00:05.827679 16290 catalog_manager.cc:4841] TS f105184c72654ad8a500398540e02870 (127.0.98.3:42337): tablet 63c1d24e7bb7406ba6a95cbacc1b1e89 (table test-table [id=f261182ee6a9443ab4a5258b9cc2ed82]) successfully deleted
I20260929 01:00:05.828894 16290 catalog_manager.cc:4841] TS ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873): tablet 63c1d24e7bb7406ba6a95cbacc1b1e89 (table test-table [id=f261182ee6a9443ab4a5258b9cc2ed82]) successfully deleted
W20260929 01:00:05.829828 16291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34505 (error 108)
I20260929 01:00:05.830714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 16497
I20260929 01:00:05.830878 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:05.830585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34505 (local address 127.0.0.1:53390)
0929 01:00:05.830708 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:05.830773 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 01:00:05.831354 16291 catalog_manager.cc:4584] TS ca028b96c255482b9b6f3d044a4aa893 (127.0.98.1:34505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63c1d24e7bb7406ba6a95cbacc1b1e89: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34505: connect: Connection refused (error 111)
I20260929 01:00:05.836349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 16633
I20260929 01:00:05.841756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 16271
I20260929 01:00:25.848907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39987
--webserver_interface=127.0.98.62
--webserver_port=37341
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:25.957860 16800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:25.958132 16800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:25.958227 16800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:25.962239 16800 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:00:25.962294 16800 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:00:25.962311 16800 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:00:25.962324 16800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:25.962378 16800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:25.962412 16800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:25.962450 16800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:25.962502 16800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:25.967814 16800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:25.968843 16800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:25.969868 16800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:25.978389 16805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:25.978389 16808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:25.978442 16806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:25.978456 16800 server_base.cc:1034] running on GCE node
I20260929 01:00:25.979224 16800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:25.980253 16800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:25.981437 16800 hybrid_clock.cc:648] HybridClock initialized: now 1790643625981423 us; error 27 us; skew 500 ppm
Sep 29 01:00:25 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:25.990401 16800 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:00:25.991314 16800 webserver.cc:460] Webserver started at http://127.0.98.62:37341/ using document root <none> and password file <none>
I20260929 01:00:25.991852 16800 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:25.991910 16800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:25.995728 16800 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:00:25.997715 16815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:25.999034 16800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:00:25.999168 16800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "45f3d8faa5614970bbb3d8c78f009511"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "8c0333661b6ba850b7c02dec58e4bd17"
server_key_iv: "6372fba098c307bb944630779c508e67"
server_key_version: "encryptionkey@0"
I20260929 01:00:25.999806 16800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:26.017668 16800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:26.020788 16800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:26.021003 16800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:26.028172 16800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39987
I20260929 01:00:26.028471 16867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39987 every 8 connection(s)
I20260929 01:00:26.029232 16800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:26.033361 16868 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:00:26.034998 16868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Bootstrap starting.
I20260929 01:00:26.035401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 16800
I20260929 01:00:26.035828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34505
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39987
--webserver_port=33347
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:00:26.052380 16868 log.cc:824] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:26.063319 16868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:00:26.063714 16868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Bootstrap complete.
I20260929 01:00:26.067690 16868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 01:00:26.068059 16868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f3d8faa5614970bbb3d8c78f009511, State: Initialized, Role: FOLLOWER
I20260929 01:00:26.068540 16868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 01:00:26.068657 16868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:26.068746 16868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:26.068837 16868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:00:26.070881 16868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 01:00:26.071211 16868 leader_election.cc:302] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f3d8faa5614970bbb3d8c78f009511; no voters:
I20260929 01:00:26.071544 16868 leader_election.cc:288] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 01:00:26.071630 16872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:00:26.071816 16872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [term 3 LEADER]: Becoming Leader. State: Replica: 45f3d8faa5614970bbb3d8c78f009511, State: Running, Role: LEADER
I20260929 01:00:26.072175 16872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } }
I20260929 01:00:26.072746 16868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:26.073081 16872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f3d8faa5614970bbb3d8c78f009511. Latest consensus state: current_term: 3 leader_uuid: "45f3d8faa5614970bbb3d8c78f009511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } } }
I20260929 01:00:26.073289 16872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:26.073468 16873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "45f3d8faa5614970bbb3d8c78f009511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f3d8faa5614970bbb3d8c78f009511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39987 } } }
I20260929 01:00:26.073648 16873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:26.073721 16876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:26.078689 16876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:26.079377 16876 catalog_manager.cc:1259] Loaded cluster ID: dede7f9ec350415abcd2021afdcea247
I20260929 01:00:26.079442 16876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:26.081892 16876 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:26.082716 16876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Loaded TSK: 0
I20260929 01:00:26.082804 16876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45f3d8faa5614970bbb3d8c78f009511: Loaded TSK: 1
F20260929 01:00:26.083621 16876 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f67f362d9ad google::LogMessage::Fail() at ??:0
@ 0x7f67f363088a google::LogMessage::SendToLog() at ??:0
@ 0x7f67f362d4de google::LogMessage::Flush() at ??:0
@ 0x7f67f362ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f67f62f118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f67f62e9f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f67fdf927bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f67fdf6c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f67fdf6d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f67fdfaedf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f67fdfcd82b std::function<>::operator()() at ??:0
@ 0x7f67fdf6cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f67fdf6df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f67fdf6a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f67fdfae640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f67f587f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f67f587fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f67f58813f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f67f5870512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f67f6eb46db start_thread at ??:0
@ 0x7f67f212b61f clone at ??:0
W20260929 01:00:26.184689 16870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:26.185048 16870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:26.185127 16870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:26.193631 16870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:26.193706 16870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:26.193835 16870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:00:26.200044 16870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33347
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:26.201138 16870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:26.202164 16870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:26.211669 16918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:26.211675 16915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:26.211815 16870 server_base.cc:1034] running on GCE node
W20260929 01:00:26.211681 16916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:26.212360 16870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:26.213019 16870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:26.214243 16870 hybrid_clock.cc:648] HybridClock initialized: now 1790643626214213 us; error 49 us; skew 500 ppm
Sep 29 01:00:26 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:26.217005 16870 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:00:26.217988 16870 webserver.cc:460] Webserver started at http://127.0.98.1:33347/ using document root <none> and password file <none>
I20260929 01:00:26.218616 16870 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:26.218677 16870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:26.222513 16870 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:26.224721 16925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:26.226014 16870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:26.226104 16870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "ca028b96c255482b9b6f3d044a4aa893"
format_stamp: "Formatted at 2026-09-29 00:59:43 on dist-test-slave-30s6"
server_key: "5217746c931a990d52cb3277cf6c77ae"
server_key_iv: "742ad1b84cf3de61c5144d68bb452f94"
server_key_version: "encryptionkey@0"
I20260929 01:00:26.226697 16870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:26.238005 16870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:26.240975 16870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:26.241165 16870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:26.241765 16870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:26.242978 16932 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 01:00:26.248323 16870 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 01:00:26.248405 16870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 01:00:26.248538 16870 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 01:00:26.250609 16870 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 01:00:26.250686 16870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:26.251017 16932 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Bootstrap starting.
I20260929 01:00:26.262537 16870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34505
I20260929 01:00:26.262629 17039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34505 every 8 connection(s)
I20260929 01:00:26.263707 16870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:26.272929 16932 log.cc:824] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:26.273002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 16870
I20260929 01:00:26.273550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33873
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39987
--webserver_port=34403
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:00:26.281739 16932 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:00:26.282168 16932 tablet_bootstrap.cc:492] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Bootstrap complete.
I20260929 01:00:26.283645 16932 ts_tablet_manager.cc:1397] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.016s
I20260929 01:00:26.286985 16932 raft_consensus.cc:348] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:26.287549 16932 raft_consensus.cc:729] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca028b96c255482b9b6f3d044a4aa893, State: Initialized, Role: FOLLOWER
I20260929 01:00:26.288079 16932 consensus_queue.cc:260] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:26.289131 16932 ts_tablet_manager.cc:1428] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260929 01:00:26.312949 17042 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:00:26.267889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:38071)
0929 01:00:26.270513 (+ 2624us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.270569 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:26.271434 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:26.312234 (+ 40800us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2415,"thread_start_us":108,"threads_started":1}
W20260929 01:00:26.313469 17040 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39987 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39987: Client connection negotiation failed: client connection to 127.0.98.62:39987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:00:26.314356 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.314023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:34263)
0929 01:00:26.314149 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.314293 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 01:00:26.315218 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.314919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:47659)
0929 01:00:26.315116 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.315165 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260929 01:00:26.315515 17040 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:00:26.392733 17044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:26.392974 17044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:26.393046 17044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:26.396970 17044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:26.397024 17044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:26.397138 17044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:00:26.402429 17044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34403
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:26.403419 17044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:26.404412 17044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:26.413722 17053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:26.413771 17050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:26.413731 17051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:26.413802 17044 server_base.cc:1034] running on GCE node
I20260929 01:00:26.414426 17044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:26.415035 17044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:26.416199 17044 hybrid_clock.cc:648] HybridClock initialized: now 1790643626416185 us; error 29 us; skew 500 ppm
Sep 29 01:00:26 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:26.418874 17044 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:00:26.419654 17044 webserver.cc:460] Webserver started at http://127.0.98.2:34403/ using document root <none> and password file <none>
I20260929 01:00:26.420161 17044 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:26.420220 17044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:26.423857 17044 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:26.425908 17060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:26.427207 17044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:26.427302 17044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "ba27d79c3b524ac48bd9d176ba54b614"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "29c2bf04f011bca2c94697ec2cd1148a"
server_key_iv: "03d8281b4222a2b40bc69f04758bec52"
server_key_version: "encryptionkey@0"
I20260929 01:00:26.427798 17044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:26.449633 17044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:26.452488 17044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:26.452642 17044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:26.453166 17044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:26.453908 17044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:26.453954 17044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:26.453994 17044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:26.454010 17044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:26.462814 17044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33873
I20260929 01:00:26.462839 17173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33873 every 8 connection(s)
I20260929 01:00:26.463780 17044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:26.466014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 17044
I20260929 01:00:26.466449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42337
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39987
--webserver_port=44295
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:00:26.467954 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.466142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:55821)
0929 01:00:26.466997 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.467142 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":141,"threads_started":1}
W20260929 01:00:26.468478 17174 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39987 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39987: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
I20260929 01:00:26.469241 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.468915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:54729)
0929 01:00:26.469073 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.469147 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 01:00:26.470002 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.469696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:36077)
0929 01:00:26.469842 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.469912 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:00:26.470219 17174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:00:26.579509 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:26.579742 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:26.579782 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:26.583611 17177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:26.583657 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:26.583714 17177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:00:26.588649 17177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44295
--tserver_master_addrs=127.0.98.62:39987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:26.589586 17177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:26.590523 17177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:26.599750 17183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:26.599779 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:26.599838 17177 server_base.cc:1034] running on GCE node
W20260929 01:00:26.599783 17185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:26.600395 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:26.600982 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:26.602159 17177 hybrid_clock.cc:648] HybridClock initialized: now 1790643626602148 us; error 29 us; skew 500 ppm
Sep 29 01:00:26 dist-test-slave-30s6 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:26.604872 17177 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:00:26.605715 17177 webserver.cc:460] Webserver started at http://127.0.98.3:44295/ using document root <none> and password file <none>
I20260929 01:00:26.606338 17177 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:26.606439 17177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:26.609995 17177 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:00:26.611994 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:26.613098 17177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:26.613240 17177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "f105184c72654ad8a500398540e02870"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-30s6"
server_key: "a82076f85703d9e05f5c2b10370afb37"
server_key_iv: "beebcdbceaac748850b8bb624559538f"
server_key_version: "encryptionkey@0"
I20260929 01:00:26.613755 17177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:26.645292 17177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:26.648669 17177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:26.648874 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:26.649479 17177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:26.650326 17177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:26.650398 17177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:26.650463 17177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:26.650518 17177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:26.659567 17177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42337
I20260929 01:00:26.659597 17305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42337 every 8 connection(s)
I20260929 01:00:26.660519 17177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:26.663650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 17177
I20260929 01:00:26.664283 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.662639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:51013)
0929 01:00:26.663272 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.663405 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":105,"threads_started":1}
W20260929 01:00:26.664799 17306 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39987 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39987: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
I20260929 01:00:26.665371 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.665172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:53647)
0929 01:00:26.665275 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.665320 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 01:00:26.665905 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.665756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:59135)
0929 01:00:26.665824 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:26.665861 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260929 01:00:26.666110 17306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:00:27.316610 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:27.316172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:33685)
0929 01:00:27.316419 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:27.316479 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260929 01:00:27.471248 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:27.470753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:35069)
0929 01:00:27.471089 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:27.471155 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":66,"threads_started":1}
I20260929 01:00:27.667052 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:27.666670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:33551)
0929 01:00:27.666898 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:27.666953 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260929 01:00:27.782517 17313 raft_consensus.cc:482] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:27.782737 17313 raft_consensus.cc:504] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:27.783803 17313 leader_election.cc:288] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
Sep 29 01:00:27 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 01:00:27 dist-test-slave-30s6 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 01:00:27.798354 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:27.784320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42337 (local address 127.0.98.1:35573)
0929 01:00:27.784628 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:27.784641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:27.784728 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:27.786362 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:27.786369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:27.787206 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:27.787534 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:27.787554 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:27.788845 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:27.788848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:27.789915 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:27.789924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:27.790725 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:27.790728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:27.790955 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:27.791654 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:27.791666 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:27.793165 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:27.795912 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:27.795917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:27.795932 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:27.796191 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:27.796558 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:27.796561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:27.796563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:27.796702 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:27.797329 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:27.797336 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:27.797623 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 01:00:27.797814 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 01:00:27.798116 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":122,"spinlock_wait_cycles":4480,"thread_start_us":94,"threads_started":1}
I20260929 01:00:27.798373 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:27.784546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35573 (local address 127.0.98.3:42337)
0929 01:00:27.784921 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:27.784939 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:27.784987 (+ 48us) server_negotiation.cc:407] Connection header received
0929 01:00:27.785616 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:27.785621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:27.785919 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:27.786287 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:27.787746 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:27.788685 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:27.790035 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:27.790590 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:27.790759 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:27.793423 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:27.793428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:27.793440 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:27.793467 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:27.795779 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:27.796320 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:27.796326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:27.796338 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:27.796450 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:27.796807 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:27.796810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:27.796812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:27.797202 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:27.797343 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:27.797928 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 01:00:27.798212 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":77,"threads_started":1}
I20260929 01:00:27.798791 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:27.784067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.98.1:50923)
0929 01:00:27.784188 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:27.784201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:27.784332 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:27.786362 (+ 2030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:27.786369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:27.786895 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:27.787534 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:27.787554 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:27.788834 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:27.788837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:27.789917 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:27.789935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:27.790721 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:27.790724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:27.790954 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:27.791598 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:27.791614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:27.793499 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:27.796840 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:27.796843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:27.796845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:27.797041 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:27.797393 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:27.797396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:27.797398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:27.797486 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:27.798065 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:27.798068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:27.798175 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 01:00:27.798336 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 01:00:27.798625 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":118}
I20260929 01:00:27.798988 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:27.784507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50923 (local address 127.0.98.2:33873)
0929 01:00:27.784814 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:27.784839 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:27.784876 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:00:27.785578 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:27.785584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:27.785919 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:27.786659 (+ 740us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:27.787707 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:27.788679 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:27.790072 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:27.790588 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:27.790759 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:27.793686 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:27.793693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:27.793708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:27.793743 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:27.796705 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:27.797154 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:27.797160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:27.797165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:27.797288 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:27.797587 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:27.797591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:27.797593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:27.797949 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:27.798081 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:27.798484 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 01:00:27.798775 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
W20260929 01:00:27.801452 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
W20260929 01:00:27.802037 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f105184c72654ad8a500398540e02870 (127.0.98.3:42337): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
I20260929 01:00:27.802114 16928 leader_election.cc:302] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca028b96c255482b9b6f3d044a4aa893; no voters: ba27d79c3b524ac48bd9d176ba54b614, f105184c72654ad8a500398540e02870
I20260929 01:00:27.802414 17313 raft_consensus.cc:2726] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:00:28.317931 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:28.317491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:49297)
0929 01:00:28.317772 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:28.317832 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
I20260929 01:00:28.472430 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:28.472023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:58281)
0929 01:00:28.472284 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:28.472343 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260929 01:00:28.668234 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:28.667853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:60721)
0929 01:00:28.668097 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:28.668152 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260929 01:00:29.319041 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:29.318615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:46931)
0929 01:00:29.318887 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:29.318947 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
I20260929 01:00:29.344302 17321 raft_consensus.cc:482] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:29.344437 17321 raft_consensus.cc:504] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:29.344869 17321 leader_election.cc:288] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
W20260929 01:00:29.345805 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f105184c72654ad8a500398540e02870 (127.0.98.3:42337): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
W20260929 01:00:29.345939 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
I20260929 01:00:29.346002 16928 leader_election.cc:302] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca028b96c255482b9b6f3d044a4aa893; no voters: ba27d79c3b524ac48bd9d176ba54b614, f105184c72654ad8a500398540e02870
I20260929 01:00:29.346248 17321 raft_consensus.cc:2726] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:00:29.473663 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:29.473202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:34209)
0929 01:00:29.473481 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:29.473551 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260929 01:00:29.669486 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:29.669046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:46591)
0929 01:00:29.669301 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:29.669367 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260929 01:00:30.320354 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:30.319925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:48931)
0929 01:00:30.320195 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:30.320253 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260929 01:00:30.474849 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:30.474451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:38525)
0929 01:00:30.474702 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:30.474759 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260929 01:00:30.670756 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:30.670345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:55689)
0929 01:00:30.670595 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:30.670654 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260929 01:00:31.321743 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:31.321308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:49913)
0929 01:00:31.321588 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:31.321648 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260929 01:00:31.476099 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:31.475664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:44707)
0929 01:00:31.475933 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:31.475997 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260929 01:00:31.671989 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:31.671549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:42371)
0929 01:00:31.671831 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:31.671894 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260929 01:00:31.719038 17330 raft_consensus.cc:482] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:31.719173 17330 raft_consensus.cc:504] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:31.719614 17330 leader_election.cc:288] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
W20260929 01:00:31.720582 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f105184c72654ad8a500398540e02870 (127.0.98.3:42337): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
W20260929 01:00:31.720732 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
I20260929 01:00:31.720794 16928 leader_election.cc:302] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca028b96c255482b9b6f3d044a4aa893; no voters: ba27d79c3b524ac48bd9d176ba54b614, f105184c72654ad8a500398540e02870
I20260929 01:00:31.721010 17330 raft_consensus.cc:2726] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:00:32.323108 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.322633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:45235)
0929 01:00:32.322931 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:32.323001 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
I20260929 01:00:32.477591 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.477033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:54597)
0929 01:00:32.477380 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:32.477438 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":99,"threads_started":1}
I20260929 01:00:32.673679 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.673029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:48107)
0929 01:00:32.673470 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:32.673544 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":114,"threads_started":1}
I20260929 01:00:33.324322 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.323909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:42893)
0929 01:00:33.324165 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.324227 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260929 01:00:33.479041 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.478470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:53251)
0929 01:00:33.478848 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.478934 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":87,"threads_started":1}
I20260929 01:00:33.675206 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.674593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:46929)
0929 01:00:33.674998 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.675060 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":113,"threads_started":1}
I20260929 01:00:34.325644 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:34.325216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:43115)
0929 01:00:34.325485 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:34.325546 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260929 01:00:34.480321 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:34.479863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:40657)
0929 01:00:34.480163 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:34.480224 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
I20260929 01:00:34.630730 17339 raft_consensus.cc:482] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:34.630908 17339 raft_consensus.cc:504] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:34.631536 17339 leader_election.cc:288] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
W20260929 01:00:34.632367 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
W20260929 01:00:34.632551 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f105184c72654ad8a500398540e02870 (127.0.98.3:42337): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
I20260929 01:00:34.632611 16928 leader_election.cc:302] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca028b96c255482b9b6f3d044a4aa893; no voters: ba27d79c3b524ac48bd9d176ba54b614, f105184c72654ad8a500398540e02870
I20260929 01:00:34.632791 17339 raft_consensus.cc:2726] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:00:34.676484 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:34.675946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:46387)
0929 01:00:34.676249 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:34.676389 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260929 01:00:35.326901 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:35.326477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:41311)
0929 01:00:35.326719 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:35.326794 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260929 01:00:35.481602 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:35.481157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:57279)
0929 01:00:35.481444 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:35.481508 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
I20260929 01:00:35.677804 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:35.677302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:45239)
0929 01:00:35.677622 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:35.677696 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
I20260929 01:00:36.328143 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:36.327715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:58481)
0929 01:00:36.327960 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:36.328046 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260929 01:00:36.482915 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:36.482493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:35647)
0929 01:00:36.482761 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:36.482822 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260929 01:00:36.679026 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:36.678615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:38893)
0929 01:00:36.678873 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:36.678933 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260929 01:00:37.329370 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:37.328911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:44195)
0929 01:00:37.329219 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:37.329275 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260929 01:00:37.484174 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:37.483740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:34829)
0929 01:00:37.484005 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:37.484061 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260929 01:00:37.680222 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:37.679765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:49567)
0929 01:00:37.680056 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:37.680121 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260929 01:00:38.330758 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:38.330305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:41901)
0929 01:00:38.330567 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:38.330629 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260929 01:00:38.485347 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:38.484917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:59873)
0929 01:00:38.485197 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:38.485254 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260929 01:00:38.681463 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:38.681064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:37253)
0929 01:00:38.681307 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:38.681366 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260929 01:00:39.331952 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:39.331510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:58975)
0929 01:00:39.331770 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:39.331849 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260929 01:00:39.406661 17354 raft_consensus.cc:482] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:39.406787 17354 raft_consensus.cc:504] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba27d79c3b524ac48bd9d176ba54b614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "f105184c72654ad8a500398540e02870" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42337 } } peers { permanent_uuid: "ca028b96c255482b9b6f3d044a4aa893" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34505 } }
I20260929 01:00:39.407135 17354 leader_election.cc:288] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873), f105184c72654ad8a500398540e02870 (127.0.98.3:42337)
W20260929 01:00:39.407941 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba27d79c3b524ac48bd9d176ba54b614 (127.0.98.2:33873): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
W20260929 01:00:39.408120 16928 leader_election.cc:341] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f105184c72654ad8a500398540e02870 (127.0.98.3:42337): Not found: Tablet not found: 63c1d24e7bb7406ba6a95cbacc1b1e89
I20260929 01:00:39.408179 16928 leader_election.cc:302] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca028b96c255482b9b6f3d044a4aa893; no voters: ba27d79c3b524ac48bd9d176ba54b614, f105184c72654ad8a500398540e02870
I20260929 01:00:39.408344 17354 raft_consensus.cc:2726] T 63c1d24e7bb7406ba6a95cbacc1b1e89 P ca028b96c255482b9b6f3d044a4aa893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:00:39.486641 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:39.486161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:48311)
0929 01:00:39.486476 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:39.486542 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":99,"threads_started":1}
I20260929 01:00:39.682857 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:39.682366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:35247)
0929 01:00:39.682666 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:39.682730 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
I20260929 01:00:40.333024 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:40.332632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:51055)
0929 01:00:40.332868 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:40.332928 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260929 01:00:40.487874 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:40.487415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:42021)
0929 01:00:40.487687 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:40.487745 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260929 01:00:40.684206 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:40.683768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.3:33175)
0929 01:00:40.684053 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:40.684112 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260929 01:00:41.334358 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:41.333886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.1:56795)
0929 01:00:41.334160 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:41.334261 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260929 01:00:41.489317 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:41.488907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39987 (local address 127.0.98.2:57915)
0929 01:00:41.489167 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:41.489224 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260929 01:00:41.667582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 16870
I20260929 01:00:41.673621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 17044
I20260929 01:00:41.679168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 17177
2026-09-29T01:00:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59449 ms)
[----------] 30 tests from SecurityITest (243170 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:41 dist-test-slave-30s6 krb5kdc[17364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:41 dist-test-slave-30s6 krb5kdc[17364](info): set up 2 sockets
Sep 29 01:00:41 dist-test-slave-30s6 krb5kdc[17364](info): commencing operation
krb5kdc: starting...
W20260929 01:00:42.730098 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:00:42 dist-test-slave-30s6 krb5kdc[17364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:00:42.871980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39163
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:42.990154 17377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:42.990502 17377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:42.990545 17377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:42.994850 17377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:42.994932 17377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:42.994952 17377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:42.994971 17377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:42.994987 17377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:43.000736 17377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:43.001929 17377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:43.003104 17377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:43.013064 17383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.013098 17382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.013127 17385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:43.013074 17377 server_base.cc:1034] running on GCE node
I20260929 01:00:43.014017 17377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:43.015355 17377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:43.016572 17377 hybrid_clock.cc:648] HybridClock initialized: now 1790643643016547 us; error 44 us; skew 500 ppm
Sep 29 01:00:43 dist-test-slave-30s6 krb5kdc[17364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:43.019713 17377 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:00:43.020651 17377 webserver.cc:460] Webserver started at http://127.0.98.62:37977/ using document root <none> and password file <none>
I20260929 01:00:43.021195 17377 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:43.021257 17377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:43.021425 17377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:43.023545 17377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "8dfaacb7939e435eaaa67861c025c9dc"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "2e0431fca6f9aecb6d3aa55d02bb68d6"
server_key_iv: "a80bbbb0044fa93fe397bd44a606253e"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.024081 17377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "8dfaacb7939e435eaaa67861c025c9dc"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "2e0431fca6f9aecb6d3aa55d02bb68d6"
server_key_iv: "a80bbbb0044fa93fe397bd44a606253e"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.028102 17377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:43.031019 17392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.032593 17377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:00:43.032773 17377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "8dfaacb7939e435eaaa67861c025c9dc"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "2e0431fca6f9aecb6d3aa55d02bb68d6"
server_key_iv: "a80bbbb0044fa93fe397bd44a606253e"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.032914 17377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:43.048316 17377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:43.052068 17377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:43.052300 17377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:43.059935 17377 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39163
I20260929 01:00:43.059971 17444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39163 every 8 connection(s)
I20260929 01:00:43.061013 17377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:43.064132 17445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:43.069362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 17377
I20260929 01:00:43.069572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:43.069912 392 external_mini_cluster.cc:1428] Setting key 042e1bd68cd384e147108f77289142fc
I20260929 01:00:43.070366 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc: Bootstrap starting.
I20260929 01:00:43.073107 17445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:43.073902 17445 log.cc:824] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:43.076121 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc: No bootstrap required, opened a new log
Sep 29 01:00:43 dist-test-slave-30s6 krb5kdc[17364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:43.080453 17445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfaacb7939e435eaaa67861c025c9dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39163 } }
I20260929 01:00:43.080705 17445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:43.080770 17445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfaacb7939e435eaaa67861c025c9dc, State: Initialized, Role: FOLLOWER
I20260929 01:00:43.081350 17445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfaacb7939e435eaaa67861c025c9dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39163 } }
I20260929 01:00:43.081537 17445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:43.081637 17445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:43.081768 17445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:43.082896 17445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfaacb7939e435eaaa67861c025c9dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39163 } }
I20260929 01:00:43.083405 17445 leader_election.cc:302] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dfaacb7939e435eaaa67861c025c9dc; no voters:
I20260929 01:00:43.083834 17445 leader_election.cc:288] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:43.083966 17450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:43.084306 17450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [term 1 LEADER]: Becoming Leader. State: Replica: 8dfaacb7939e435eaaa67861c025c9dc, State: Running, Role: LEADER
I20260929 01:00:43.084709 17450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfaacb7939e435eaaa67861c025c9dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39163 } }
I20260929 01:00:43.085369 17445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:43.086587 17451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dfaacb7939e435eaaa67861c025c9dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfaacb7939e435eaaa67861c025c9dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39163 } } }
I20260929 01:00:43.086730 17451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:43.087211 17452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dfaacb7939e435eaaa67861c025c9dc. Latest consensus state: current_term: 1 leader_uuid: "8dfaacb7939e435eaaa67861c025c9dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfaacb7939e435eaaa67861c025c9dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39163 } } }
I20260929 01:00:43.087342 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.071540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.98.62:39163)
0929 01:00:43.072074 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:43.072091 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:43.072138 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:00:43.072837 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:43.072856 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:43.073426 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:43.073803 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:43.074808 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.075809 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.076789 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.077309 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.077490 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.080526 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:43.080558 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:43.080574 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:43.080607 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:43.083239 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.084106 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.084115 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.084123 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.084282 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.084632 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.084635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.084637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.085194 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:43.085409 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:43.085730 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 01:00:43.086087 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":173,"threads_started":1}
I20260929 01:00:43.087368 17452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc [sys.catalog]: This master's current role is: LEADER
W20260929 01:00:43.091003 17465 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:00:43.091100 17465 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:00:43.091236 17459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:43.091985 17459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:43.098284 17459 catalog_manager.cc:1347] Generated new cluster ID: be7c2a03902e4d55a44b929aac726a42
I20260929 01:00:43.098464 17459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:43.109330 17459 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:43.110472 17459 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:43.123036 17459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc: Generated new TSK 0
I20260929 01:00:43.123908 17459 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:00:43.167330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39163
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:43.283401 17471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:43.283686 17471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:43.283761 17471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:43.288055 17471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:43.288143 17471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:43.288236 17471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:00:43.293573 17471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:43.294678 17471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:43.295740 17471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:43.305195 17477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.305205 17479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.305531 17476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:43.305677 17471 server_base.cc:1034] running on GCE node
I20260929 01:00:43.306087 17471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:43.306730 17471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:43.307915 17471 hybrid_clock.cc:648] HybridClock initialized: now 1790643643307901 us; error 30 us; skew 500 ppm
Sep 29 01:00:43 dist-test-slave-30s6 krb5kdc[17364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:43.310786 17471 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:00:43.311671 17471 webserver.cc:460] Webserver started at http://127.0.98.1:39029/ using document root <none> and password file <none>
I20260929 01:00:43.312220 17471 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:43.312281 17471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:43.312448 17471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:43.314607 17471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "5207ce1f19b046b791fe911885086ab5"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "bd910cc389022f27916f10f3de45ab2a"
server_key_iv: "5cc18ec3040abf15521a484a6b3f65c6"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.315177 17471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5207ce1f19b046b791fe911885086ab5"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "bd910cc389022f27916f10f3de45ab2a"
server_key_iv: "5cc18ec3040abf15521a484a6b3f65c6"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.319007 17471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:00:43.321388 17486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.322685 17471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:43.322821 17471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "5207ce1f19b046b791fe911885086ab5"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "bd910cc389022f27916f10f3de45ab2a"
server_key_iv: "5cc18ec3040abf15521a484a6b3f65c6"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.322933 17471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:43.336925 17471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:43.340415 17471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:43.340636 17471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:43.341228 17471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:43.342067 17471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:43.342136 17471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.342303 17471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:43.342353 17471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.351089 17471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46565
I20260929 01:00:43.351114 17599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46565 every 8 connection(s)
I20260929 01:00:43.351965 17471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:43.354321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 17471
I20260929 01:00:43.354498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:43.354876 392 external_mini_cluster.cc:1428] Setting key 97bb26e9a328050dbb453ad9f46f8100
Sep 29 01:00:43 dist-test-slave-30s6 krb5kdc[17364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:00:43.367027 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.353660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48429 (local address 127.0.98.62:39163)
0929 01:00:43.353824 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:43.353829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:43.354843 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 01:00:43.355935 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:43.355939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:43.355983 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:43.356086 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:43.357718 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.358356 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.359923 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.360373 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.360501 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.363306 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:43.363310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:43.363314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:43.363345 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:43.364880 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.365440 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.365446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.365448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.365531 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.365883 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.365888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.365890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.366101 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:43.366232 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:43.366734 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 01:00:43.366860 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:00:43.368042 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.354052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39163 (local address 127.0.98.1:48429)
0929 01:00:43.354635 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:43.354697 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:43.355731 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:43.356320 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:43.356328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:43.356842 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:43.357558 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.357579 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.358488 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.358493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.359793 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.359803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.360521 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.360524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.360658 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.361399 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:43.361425 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:43.363129 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:43.365018 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:43.365024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:43.365041 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:43.365332 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:43.365633 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:43.365636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:43.365638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:43.365784 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:43.366234 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:43.366242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:43.366494 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 01:00:43.366695 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 01:00:43.367147 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":175,"threads_started":1}
I20260929 01:00:43.369463 17600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39163
I20260929 01:00:43.369767 17600 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:43.370433 17600 heartbeater.cc:511] Master 127.0.98.62:39163 requested a full tablet report, sending...
I20260929 01:00:43.372195 17407 ts_manager.cc:194] Registered new tserver with Master: 5207ce1f19b046b791fe911885086ab5 (127.0.98.1:46565)
I20260929 01:00:43.373385 17407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:00:43.386428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39163
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:43.498947 17605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:43.499204 17605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:43.499246 17605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:43.503265 17605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:43.503315 17605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:43.503371 17605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:00:43.508378 17605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:43.509388 17605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:43.510404 17605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:43.519711 17613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.519826 17611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.519711 17610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:43.520326 17605 server_base.cc:1034] running on GCE node
I20260929 01:00:43.520781 17605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:43.521469 17605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:43.522680 17605 hybrid_clock.cc:648] HybridClock initialized: now 1790643643522643 us; error 48 us; skew 500 ppm
Sep 29 01:00:43 dist-test-slave-30s6 krb5kdc[17364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:43.525600 17605 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:00:43.526616 17605 webserver.cc:460] Webserver started at http://127.0.98.2:34429/ using document root <none> and password file <none>
I20260929 01:00:43.527257 17605 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:43.527324 17605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:43.527493 17605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:43.529583 17605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "78fb04a9270d4ae295851496ac28c690"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "30a06316c0f47bbf31b22d2cf1e1f0c8"
server_key_iv: "f6eae3937f6b9b9863b171bb070875cc"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.530119 17605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78fb04a9270d4ae295851496ac28c690"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "30a06316c0f47bbf31b22d2cf1e1f0c8"
server_key_iv: "f6eae3937f6b9b9863b171bb070875cc"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.534013 17605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:00:43.536823 17620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.538112 17605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:00:43.538290 17605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "78fb04a9270d4ae295851496ac28c690"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "30a06316c0f47bbf31b22d2cf1e1f0c8"
server_key_iv: "f6eae3937f6b9b9863b171bb070875cc"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.538448 17605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:43.556116 17605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:43.559165 17605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:43.559373 17605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:43.559940 17605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:43.560734 17605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:43.560806 17605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.560876 17605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:43.560930 17605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.569805 17605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41509
I20260929 01:00:43.569831 17733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41509 every 8 connection(s)
I20260929 01:00:43.570739 17605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:43.572906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 17605
I20260929 01:00:43.573035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:43.573376 392 external_mini_cluster.cc:1428] Setting key 1a8a493ceade51951b980706dbcbdae2
Sep 29 01:00:43 dist-test-slave-30s6 krb5kdc[17364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:00:43.584605 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.572562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56909 (local address 127.0.98.62:39163)
0929 01:00:43.572706 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:43.572709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:43.573750 (+ 1041us) server_negotiation.cc:407] Connection header received
0929 01:00:43.574961 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:43.574964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:43.575016 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:43.575096 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:43.576237 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.576798 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.577929 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.578356 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.578476 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.580822 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:43.580829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:43.580834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:43.580868 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:43.582579 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.583048 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.583054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.583057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.583159 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.583477 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.583481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.583483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.583740 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:43.583847 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:43.584343 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 01:00:43.584438 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:00:43.585435 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.573060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39163 (local address 127.0.98.2:56909)
0929 01:00:43.573607 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:43.573649 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:43.574645 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:43.575292 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:43.575299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:43.575645 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:43.576099 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.576112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.576917 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.576920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.577804 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.577813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.578498 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.578501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.578654 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.579260 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:43.579276 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:43.580658 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:43.582685 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:43.582689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:43.582701 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:43.582940 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:43.583262 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:43.583264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:43.583265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:43.583385 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:43.583874 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:43.583880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:43.584119 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 01:00:43.584281 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 01:00:43.584526 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:00:43.586759 17734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39163
I20260929 01:00:43.587013 17734 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:43.587589 17734 heartbeater.cc:511] Master 127.0.98.62:39163 requested a full tablet report, sending...
I20260929 01:00:43.588726 17407 ts_manager.cc:194] Registered new tserver with Master: 78fb04a9270d4ae295851496ac28c690 (127.0.98.2:41509)
I20260929 01:00:43.589279 17407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:00:43.601485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39163
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:43.711162 17739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:43.711427 17739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:43.711469 17739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:43.715481 17739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:43.715540 17739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:43.715603 17739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:00:43.721024 17739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:43.722009 17739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:43.723026 17739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:43.732968 17747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:43.733140 17739 server_base.cc:1034] running on GCE node
W20260929 01:00:43.733031 17745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.732978 17744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:43.733675 17739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:43.734614 17739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:43.736079 17739 hybrid_clock.cc:648] HybridClock initialized: now 1790643643736062 us; error 74 us; skew 500 ppm
Sep 29 01:00:43 dist-test-slave-30s6 krb5kdc[17364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:43.738903 17739 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:00:43.739811 17739 webserver.cc:460] Webserver started at http://127.0.98.3:41269/ using document root <none> and password file <none>
I20260929 01:00:43.740386 17739 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:43.740476 17739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:43.740705 17739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:43.743093 17739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "71e83180e5ed451bb99f8d7960cd90f1"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "7bcf06f2cd8fbf4b72674c739b7f5455"
server_key_iv: "2c40bee8410bb2188a430275202fb562"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.743784 17739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71e83180e5ed451bb99f8d7960cd90f1"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "7bcf06f2cd8fbf4b72674c739b7f5455"
server_key_iv: "2c40bee8410bb2188a430275202fb562"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.747426 17739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 01:00:43.749698 17754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.750931 17739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:00:43.751022 17739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "71e83180e5ed451bb99f8d7960cd90f1"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-30s6"
server_key: "7bcf06f2cd8fbf4b72674c739b7f5455"
server_key_iv: "2c40bee8410bb2188a430275202fb562"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.751098 17739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:43.766937 17739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:43.769704 17739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:43.769858 17739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:43.770452 17739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:43.771224 17739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:43.771268 17739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.771308 17739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:43.771324 17739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.780344 17739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42087
I20260929 01:00:43.780366 17867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42087 every 8 connection(s)
I20260929 01:00:43.781208 17739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:43.787719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 17739
I20260929 01:00:43.787869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:43.788166 392 external_mini_cluster.cc:1428] Setting key 51e52cd8e7a59561584d6659b1557e7f
Sep 29 01:00:43 dist-test-slave-30s6 krb5kdc[17364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:43.795647 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.783156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35235 (local address 127.0.98.62:39163)
0929 01:00:43.783303 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:43.783307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:43.784148 (+ 841us) server_negotiation.cc:407] Connection header received
0929 01:00:43.784971 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:43.784973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:43.785038 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:43.785168 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:43.786543 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.787140 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.788574 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.788818 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.788945 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.791784 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:43.791795 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:43.791798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:43.791823 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:43.793413 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.793964 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.793971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.793976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.794066 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.794485 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.794489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.794491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.794687 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:43.794818 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:43.795346 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 01:00:43.795468 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:00:43.796514 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.783505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39163 (local address 127.0.98.3:35235)
0929 01:00:43.783950 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:43.784001 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:43.784780 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:43.785306 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:43.785312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:43.785661 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:43.786393 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.786413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.787259 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.787262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.788437 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.788447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.788981 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.788985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.789184 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.789852 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:43.789870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:43.791606 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:43.793548 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:43.793555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:43.793570 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:43.793850 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:43.794204 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:43.794208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:43.794210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:43.794366 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:43.794813 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:43.794821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:43.795090 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 01:00:43.795285 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 01:00:43.795641 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260929 01:00:43.797736 17868 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39163
I20260929 01:00:43.797968 17868 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:43.798507 17868 heartbeater.cc:511] Master 127.0.98.62:39163 requested a full tablet report, sending...
I20260929 01:00:43.799556 17407 ts_manager.cc:194] Registered new tserver with Master: 71e83180e5ed451bb99f8d7960cd90f1 (127.0.98.3:42087)
I20260929 01:00:43.800177 17407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35235
I20260929 01:00:43.801817 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:43.811295 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.803845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.0.98.62:39163)
0929 01:00:43.803978 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:43.803982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:43.804087 (+ 105us) server_negotiation.cc:407] Connection header received
0929 01:00:43.804190 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:43.804192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:43.804228 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:43.804300 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:43.804994 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.805540 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.806460 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.806701 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.806789 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.808058 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:43.808061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:43.808064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:43.808088 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:43.809822 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.810259 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.810261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.810263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.810339 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:43.810573 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:43.810575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:43.810576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:43.810799 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:43.810908 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:43.811076 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 01:00:43.811170 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:00:43.814092 17407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:43.817131 17407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:43.831264 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.825243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46565 (local address 127.0.0.1:37318)
0929 01:00:43.825640 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:43.825695 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:43.826144 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:43.826989 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:43.826994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:43.827018 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:43.827152 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.827158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.828346 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.828352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.829600 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.829619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.830684 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.830688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.830801 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.830835 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:00:43.830981 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 01:00:43.831106 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260929 01:00:43.831274 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.825410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.0.98.1:46565)
0929 01:00:43.826472 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:43.826487 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:43.826512 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:00:43.826591 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:43.826599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:43.826827 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:43.826997 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:43.827277 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.828223 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.829744 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.830561 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.830684 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.830835 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:43.831012 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 01:00:43.831106 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":75,"threads_started":1}
I20260929 01:00:43.831830 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.826000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42087 (local address 127.0.0.1:36252)
0929 01:00:43.826578 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:43.826594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:43.826682 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:43.827642 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:43.827645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:43.827661 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:43.827736 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.827741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.829087 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.829090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.830230 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.830241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.831454 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.831457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.831541 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.831560 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 01:00:43.831639 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 01:00:43.831700 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":82,"threads_started":1}
I20260929 01:00:43.831876 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.826078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.98.2:41509)
0929 01:00:43.826926 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:43.826940 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:43.826961 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:00:43.827029 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:43.827035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:43.827246 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:43.827390 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:43.827610 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.828689 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.829992 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.830781 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.830882 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.831013 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:43.831641 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 01:00:43.831734 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":130,"threads_started":1}
I20260929 01:00:43.832190 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.825601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41509 (local address 127.0.0.1:57722)
0929 01:00:43.825870 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:43.825887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:43.826007 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:43.827377 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:43.827380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:43.827397 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:43.827497 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.827503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.828841 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.828845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.829883 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:43.829890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.831075 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.831079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:43.831171 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.831209 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 01:00:43.831275 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 01:00:43.831343 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260929 01:00:43.832196 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:43.826472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.0.98.3:42087)
0929 01:00:43.826994 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:43.827006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:43.827028 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:00:43.827102 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:43.827106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:43.827326 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:43.827471 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:43.827869 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.828950 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.830365 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:43.831155 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:43.831271 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:43.831398 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:43.831676 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 01:00:43.831783 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":143,"threads_started":1}
I20260929 01:00:43.834687 17668 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb82d4975bf4807acf72998d499afa0 (DEFAULT_TABLE table=test-table [id=20d787864b764fe881d70659f7af0abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:43.834695 17534 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb82d4975bf4807acf72998d499afa0 (DEFAULT_TABLE table=test-table [id=20d787864b764fe881d70659f7af0abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:43.835071 17802 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb82d4975bf4807acf72998d499afa0 (DEFAULT_TABLE table=test-table [id=20d787864b764fe881d70659f7af0abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:43.835711 17668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb82d4975bf4807acf72998d499afa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:43.835739 17534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb82d4975bf4807acf72998d499afa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:43.835937 17802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb82d4975bf4807acf72998d499afa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:43.843389 17885 tablet_bootstrap.cc:492] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Bootstrap starting.
I20260929 01:00:43.845623 17886 tablet_bootstrap.cc:492] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5: Bootstrap starting.
I20260929 01:00:43.846573 17885 tablet_bootstrap.cc:654] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:43.846566 17887 tablet_bootstrap.cc:492] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Bootstrap starting.
I20260929 01:00:43.847471 17885 log.cc:824] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:43.847994 17886 tablet_bootstrap.cc:654] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:43.848917 17887 tablet_bootstrap.cc:654] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:43.849285 17886 log.cc:824] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:43.850049 17887 log.cc:824] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:43.850754 17885 tablet_bootstrap.cc:492] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: No bootstrap required, opened a new log
I20260929 01:00:43.850971 17885 ts_tablet_manager.cc:1397] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 01:00:43.851264 17886 tablet_bootstrap.cc:492] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5: No bootstrap required, opened a new log
I20260929 01:00:43.851475 17886 ts_tablet_manager.cc:1397] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:00:43.852089 17887 tablet_bootstrap.cc:492] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: No bootstrap required, opened a new log
I20260929 01:00:43.852265 17887 ts_tablet_manager.cc:1397] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:43.854624 17886 raft_consensus.cc:348] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:43.854888 17886 raft_consensus.cc:374] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:43.854943 17886 raft_consensus.cc:729] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5207ce1f19b046b791fe911885086ab5, State: Initialized, Role: FOLLOWER
I20260929 01:00:43.854918 17885 raft_consensus.cc:348] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:43.855191 17885 raft_consensus.cc:374] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:43.855245 17885 raft_consensus.cc:729] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78fb04a9270d4ae295851496ac28c690, State: Initialized, Role: FOLLOWER
I20260929 01:00:43.855433 17886 consensus_queue.cc:260] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:43.856137 17600 heartbeater.cc:503] Master 127.0.98.62:39163 was elected leader, sending a full tablet report...
I20260929 01:00:43.855976 17887 raft_consensus.cc:348] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:43.856220 17887 raft_consensus.cc:374] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:43.856268 17887 raft_consensus.cc:729] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e83180e5ed451bb99f8d7960cd90f1, State: Initialized, Role: FOLLOWER
I20260929 01:00:43.856496 17886 ts_tablet_manager.cc:1428] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 01:00:43.856631 17885 consensus_queue.cc:260] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:43.856657 17887 consensus_queue.cc:260] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:43.857257 17734 heartbeater.cc:503] Master 127.0.98.62:39163 was elected leader, sending a full tablet report...
I20260929 01:00:43.857554 17885 ts_tablet_manager.cc:1428] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 01:00:43.857554 17887 ts_tablet_manager.cc:1428] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 01:00:43.858261 17868 heartbeater.cc:503] Master 127.0.98.62:39163 was elected leader, sending a full tablet report...
W20260929 01:00:44.032261 17869 tablet.cc:2369] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:44.036144 17891 raft_consensus.cc:482] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:44.036394 17891 raft_consensus.cc:504] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:44.037689 17891 leader_election.cc:288] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e83180e5ed451bb99f8d7960cd90f1 (127.0.98.3:42087), 78fb04a9270d4ae295851496ac28c690 (127.0.98.2:41509)
I20260929 01:00:44.041671 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.038046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42723 (local address 127.0.98.3:42087)
0929 01:00:44.038325 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.038333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.038365 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:00:44.038543 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.038548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.038597 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.038687 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:44.038956 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.039462 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.040653 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.041059 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.041194 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.041280 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.041450 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.041536 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260929 01:00:44.041695 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.037940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42087 (local address 127.0.98.1:42723)
0929 01:00:44.038290 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.038305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:44.038429 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:44.038699 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:44.038702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:44.038718 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:44.038843 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.038849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.039587 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.039590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.040505 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.040515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.041198 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.041201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.041309 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.041340 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:00:44.041430 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 01:00:44.041549 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":81,"thread_start_us":120,"threads_started":1}
I20260929 01:00:44.042012 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.038050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57515 (local address 127.0.98.2:41509)
0929 01:00:44.038229 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.038234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.038521 (+ 287us) server_negotiation.cc:407] Connection header received
0929 01:00:44.038659 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.038662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.038714 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.038807 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:44.039102 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.039631 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.040765 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.041262 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.041421 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.041492 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.041756 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.041885 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:00:44.042124 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.037956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41509 (local address 127.0.98.1:57515)
0929 01:00:44.038419 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.038432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:44.038563 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:44.038818 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:44.038821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:44.038838 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:44.038974 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.038980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.039750 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.039756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.040621 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.040630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.041421 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.041425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.041601 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.041633 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:00:44.041882 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 01:00:44.041959 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":22,"thread_start_us":99,"threads_started":1}
I20260929 01:00:44.042735 17822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb82d4975bf4807acf72998d499afa0" candidate_uuid: "5207ce1f19b046b791fe911885086ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e83180e5ed451bb99f8d7960cd90f1" is_pre_election: true
I20260929 01:00:44.042827 17688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb82d4975bf4807acf72998d499afa0" candidate_uuid: "5207ce1f19b046b791fe911885086ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78fb04a9270d4ae295851496ac28c690" is_pre_election: true
I20260929 01:00:44.043094 17822 raft_consensus.cc:2454] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5207ce1f19b046b791fe911885086ab5 in term 0.
I20260929 01:00:44.043288 17688 raft_consensus.cc:2454] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5207ce1f19b046b791fe911885086ab5 in term 0.
I20260929 01:00:44.043638 17489 leader_election.cc:302] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5207ce1f19b046b791fe911885086ab5, 71e83180e5ed451bb99f8d7960cd90f1; no voters:
I20260929 01:00:44.043951 17891 raft_consensus.cc:2781] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:44.044031 17891 raft_consensus.cc:482] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:44.044065 17891 raft_consensus.cc:3037] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:44.045185 17891 raft_consensus.cc:504] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:44.045557 17891 leader_election.cc:288] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [CANDIDATE]: Term 1 election: Requested vote from peers 71e83180e5ed451bb99f8d7960cd90f1 (127.0.98.3:42087), 78fb04a9270d4ae295851496ac28c690 (127.0.98.2:41509)
I20260929 01:00:44.045964 17822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb82d4975bf4807acf72998d499afa0" candidate_uuid: "5207ce1f19b046b791fe911885086ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e83180e5ed451bb99f8d7960cd90f1"
I20260929 01:00:44.045964 17688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb82d4975bf4807acf72998d499afa0" candidate_uuid: "5207ce1f19b046b791fe911885086ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78fb04a9270d4ae295851496ac28c690"
I20260929 01:00:44.046098 17688 raft_consensus.cc:3037] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:44.046098 17822 raft_consensus.cc:3037] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:44.047516 17688 raft_consensus.cc:2454] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5207ce1f19b046b791fe911885086ab5 in term 1.
I20260929 01:00:44.047518 17822 raft_consensus.cc:2454] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5207ce1f19b046b791fe911885086ab5 in term 1.
I20260929 01:00:44.047891 17488 leader_election.cc:302] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5207ce1f19b046b791fe911885086ab5, 78fb04a9270d4ae295851496ac28c690; no voters:
I20260929 01:00:44.048194 17891 raft_consensus.cc:2781] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:44.048491 17891 raft_consensus.cc:686] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 5207ce1f19b046b791fe911885086ab5, State: Running, Role: LEADER
I20260929 01:00:44.048840 17891 consensus_queue.cc:237] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } }
I20260929 01:00:44.052340 17408 catalog_manager.cc:5495] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5207ce1f19b046b791fe911885086ab5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5207ce1f19b046b791fe911885086ab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5207ce1f19b046b791fe911885086ab5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:44.068321 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.064296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.0.98.1:46565)
0929 01:00:44.064450 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.064455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.064473 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:00:44.064566 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.064569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.064635 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.064758 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:44.064986 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.065799 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.066719 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.067068 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.067168 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.067454 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:44.067962 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:44.068074 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.068121 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.068184 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 01:00:44.071871 17735 tablet.cc:2369] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:44.074468 17688 raft_consensus.cc:1261] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 1 FOLLOWER]: Refusing update from remote peer 5207ce1f19b046b791fe911885086ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:44.074469 17822 raft_consensus.cc:1261] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 1 FOLLOWER]: Refusing update from remote peer 5207ce1f19b046b791fe911885086ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:44.075332 17891 consensus_queue.cc:1035] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e83180e5ed451bb99f8d7960cd90f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:44.075582 17896 consensus_queue.cc:1035] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78fb04a9270d4ae295851496ac28c690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:44.083411 17900 mvcc.cc:205] Tried to move back new op lower bound from 7334476366121308160 to 7334476366026817536. Current Snapshot: MvccSnapshot[applied={T|T < 7334476366121308160}]
I20260929 01:00:44.090853 17407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:44.091050 17407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:44.093624 17407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8dfaacb7939e435eaaa67861c025c9dc: Sending DeleteTablet for 3 replicas of tablet 8cb82d4975bf4807acf72998d499afa0
I20260929 01:00:44.094491 17534 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb82d4975bf4807acf72998d499afa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37318
I20260929 01:00:44.094507 17668 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb82d4975bf4807acf72998d499afa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57722
I20260929 01:00:44.094563 17802 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb82d4975bf4807acf72998d499afa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36252
I20260929 01:00:44.095018 17910 tablet_replica.cc:330] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5: stopping tablet replica
I20260929 01:00:44.095362 17910 raft_consensus.cc:2229] T 8cb82d4975bf4807acf72998d499afa0 P 5207ce1f19b046b791fe911885086ab5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:44.095341 17909 tablet_replica.cc:330] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: stopping tablet replica
I20260929 01:00:44.095712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 17471
I20260929 01:00:44.096695 17911 tablet_replica.cc:330] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: stopping tablet replica
I20260929 01:00:44.097062 17911 raft_consensus.cc:2229] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:44.097379 17911 raft_consensus.cc:2258] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:44.098361 17909 raft_consensus.cc:2229] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:44.098675 17909 raft_consensus.cc:2258] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:44.098838 17911 ts_tablet_manager.cc:1905] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:44.100136 17909 ts_tablet_manager.cc:1905] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:44.101131 17911 ts_tablet_manager.cc:1918] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:44.101238 17911 log.cc:1196] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/8cb82d4975bf4807acf72998d499afa0
I20260929 01:00:44.101514 17911 ts_tablet_manager.cc:1939] T 8cb82d4975bf4807acf72998d499afa0 P 71e83180e5ed451bb99f8d7960cd90f1: Deleting consensus metadata
I20260929 01:00:44.102931 17395 catalog_manager.cc:4841] TS 71e83180e5ed451bb99f8d7960cd90f1 (127.0.98.3:42087): tablet 8cb82d4975bf4807acf72998d499afa0 (table test-table [id=20d787864b764fe881d70659f7af0abc]) successfully deleted
I20260929 01:00:44.102963 17909 ts_tablet_manager.cc:1918] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:44.103062 17909 log.cc:1196] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/8cb82d4975bf4807acf72998d499afa0
W20260929 01:00:44.103216 17395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46565 (error 108)
I20260929 01:00:44.103406 17909 ts_tablet_manager.cc:1939] T 8cb82d4975bf4807acf72998d499afa0 P 78fb04a9270d4ae295851496ac28c690: Deleting consensus metadata
I20260929 01:00:44.103434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 17605
I20260929 01:00:44.107570 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.107294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46565 (local address 127.0.0.1:37342)
0929 01:00:44.107428 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.107499 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:00:44.107774 17395 catalog_manager.cc:4584] TS 5207ce1f19b046b791fe911885086ab5 (127.0.98.1:46565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cb82d4975bf4807acf72998d499afa0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46565: connect: Connection refused (error 111)
I20260929 01:00:44.110661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 17739
I20260929 01:00:44.111476 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.110742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41509 (local address 127.0.0.1:57730)
0929 01:00:44.111318 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.111380 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481}
I20260929 01:00:44.116207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 17377
2026-09-29T01:00:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2437 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T01:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:44Z Disabled control of system clock
I20260929 01:00:44.146391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33981
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:00:44.257948 17916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:44.258239 17916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:44.258275 17916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:44.262331 17916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:44.262382 17916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:44.262398 17916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:44.262445 17916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:44.262476 17916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:44.267119 17916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:44.268195 17916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:44.269582 17916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:44.278263 17921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:44.278290 17922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:44.278281 17916 server_base.cc:1034] running on GCE node
W20260929 01:00:44.278263 17924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:44.279032 17916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:44.280143 17916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:44.281351 17916 hybrid_clock.cc:648] HybridClock initialized: now 1790643644281336 us; error 26 us; skew 500 ppm
I20260929 01:00:44.283420 17916 webserver.cc:460] Webserver started at http://127.0.98.62:38077/ using document root <none> and password file <none>
I20260929 01:00:44.284024 17916 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:44.284132 17916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:44.284379 17916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:44.286571 17916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "2ba7d2886d1d40cfa44963ad28e75ccd"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "0f13faa81c5280ad0ee7716c16d3660e"
server_key_iv: "f9663784955f01b55644f641b2f0c19a"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.287184 17916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ba7d2886d1d40cfa44963ad28e75ccd"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "0f13faa81c5280ad0ee7716c16d3660e"
server_key_iv: "f9663784955f01b55644f641b2f0c19a"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.291028 17916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:00:44.293466 17930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.294701 17916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:44.294842 17916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "2ba7d2886d1d40cfa44963ad28e75ccd"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "0f13faa81c5280ad0ee7716c16d3660e"
server_key_iv: "f9663784955f01b55644f641b2f0c19a"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.294948 17916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:44.311918 17916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:44.312618 17916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:44.312819 17916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:44.319708 17916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33981
I20260929 01:00:44.319727 17982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33981 every 8 connection(s)
I20260929 01:00:44.320640 17916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:44.321904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 17916
I20260929 01:00:44.322039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:44.322381 392 external_mini_cluster.cc:1428] Setting key 2539d0823678aa8724cd5b463cf94c24
I20260929 01:00:44.324186 17983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:44.329742 17983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd: Bootstrap starting.
I20260929 01:00:44.331499 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.323854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.0.98.62:33981)
0929 01:00:44.324560 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.324575 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.324610 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:00:44.325263 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.325279 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.325660 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.325990 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:44.326449 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.327380 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.328251 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.328714 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.328884 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.329770 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:44.329775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:44.329787 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:44.330271 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:44.330325 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:44.330345 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:44.330451 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.330563 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.330818 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":155,"threads_started":1}
I20260929 01:00:44.332180 17983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:44.332931 17983 log.cc:824] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:44.335359 17983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd: No bootstrap required, opened a new log
I20260929 01:00:44.339607 17983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba7d2886d1d40cfa44963ad28e75ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } }
I20260929 01:00:44.339856 17983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:44.339916 17983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba7d2886d1d40cfa44963ad28e75ccd, State: Initialized, Role: FOLLOWER
I20260929 01:00:44.340360 17983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba7d2886d1d40cfa44963ad28e75ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } }
I20260929 01:00:44.340512 17983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:44.340575 17983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:44.340668 17983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:44.341557 17983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba7d2886d1d40cfa44963ad28e75ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } }
I20260929 01:00:44.341892 17983 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba7d2886d1d40cfa44963ad28e75ccd; no voters:
I20260929 01:00:44.342250 17983 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:44.342368 17988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:44.342594 17988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [term 1 LEADER]: Becoming Leader. State: Replica: 2ba7d2886d1d40cfa44963ad28e75ccd, State: Running, Role: LEADER
I20260929 01:00:44.342905 17988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba7d2886d1d40cfa44963ad28e75ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } }
I20260929 01:00:44.343442 17983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:44.344498 17989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba7d2886d1d40cfa44963ad28e75ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba7d2886d1d40cfa44963ad28e75ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } } }
I20260929 01:00:44.344630 17989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:44.344902 17990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba7d2886d1d40cfa44963ad28e75ccd. Latest consensus state: current_term: 1 leader_uuid: "2ba7d2886d1d40cfa44963ad28e75ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba7d2886d1d40cfa44963ad28e75ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } } }
I20260929 01:00:44.345000 17990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:44.345081 17997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:44.348269 17997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:44.354480 17997 catalog_manager.cc:1347] Generated new cluster ID: b7f2bb27c26345c49597c00c2c31cba6
I20260929 01:00:44.354573 17997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:44.373330 17997 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:44.374387 17997 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:44.385556 17997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd: Generated new TSK 0
I20260929 01:00:44.386407 17997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:44.389075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:00:44.510888 18007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:44.511157 18007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:44.511237 18007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:44.515221 18007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:44.515297 18007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:44.515389 18007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:00:44.519999 18007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:44.521034 18007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:44.522454 18007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:44.531869 18012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:44.531900 18007 server_base.cc:1034] running on GCE node
W20260929 01:00:44.531906 18013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:44.532049 18015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:44.532549 18007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:44.533180 18007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:44.534364 18007 hybrid_clock.cc:648] HybridClock initialized: now 1790643644534346 us; error 28 us; skew 500 ppm
I20260929 01:00:44.536335 18007 webserver.cc:460] Webserver started at http://127.0.98.1:39063/ using document root <none> and password file <none>
I20260929 01:00:44.536921 18007 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:44.537015 18007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:44.537248 18007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:44.539392 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "eec67310f79745bc8180434635e21392"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "f03aee920f18a03ed60b78c12d1023ac"
server_key_iv: "0735042350f366ea4051d83da7123afd"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.539951 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eec67310f79745bc8180434635e21392"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "f03aee920f18a03ed60b78c12d1023ac"
server_key_iv: "0735042350f366ea4051d83da7123afd"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.543705 18007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:44.546464 18021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.547724 18007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:44.547863 18007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "eec67310f79745bc8180434635e21392"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "f03aee920f18a03ed60b78c12d1023ac"
server_key_iv: "0735042350f366ea4051d83da7123afd"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.547983 18007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:44.567938 18007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:44.568624 18007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:44.568825 18007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:44.569403 18007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:44.570384 18007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:44.570452 18007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.570524 18007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:44.570575 18007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.579596 18007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39235
I20260929 01:00:44.579618 18134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39235 every 8 connection(s)
I20260929 01:00:44.580478 18007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:44.585891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 18007
I20260929 01:00:44.586041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:44.586392 392 external_mini_cluster.cc:1428] Setting key da10c4b825328a14fc2152eb073a0986
I20260929 01:00:44.588243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:00:44.589995 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.582335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51987 (local address 127.0.98.62:33981)
0929 01:00:44.582475 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.582478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.583491 (+ 1013us) server_negotiation.cc:407] Connection header received
0929 01:00:44.584926 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.584929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.584992 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.585094 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:44.586001 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.586681 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.588009 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.588255 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.588363 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.589473 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:44.589477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:44.589485 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:44.589497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:44.589516 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:44.589523 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:44.589602 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.589788 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.589853 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:00:44.590837 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.582699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33981 (local address 127.0.98.1:51987)
0929 01:00:44.583343 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.583382 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:44.584702 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:44.585278 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:44.585285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:44.585373 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:44.585857 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.585869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.586810 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.586814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.587892 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.587901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.588413 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.588418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.588582 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.589227 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:44.589228 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:44.589266 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:44.589325 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:44.589604 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:44.589616 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:44.589624 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:00:44.589775 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 01:00:44.590047 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260929 01:00:44.592016 18135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33981
I20260929 01:00:44.592249 18135 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:44.592777 18135 heartbeater.cc:511] Master 127.0.98.62:33981 requested a full tablet report, sending...
I20260929 01:00:44.594285 17947 ts_manager.cc:194] Registered new tserver with Master: eec67310f79745bc8180434635e21392 (127.0.98.1:39235)
I20260929 01:00:44.595391 17947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51987
W20260929 01:00:44.699536 18138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:44.699798 18138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:44.699868 18138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:44.703909 18138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:44.703962 18138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:44.704051 18138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:00:44.708846 18138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:44.709947 18138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:44.711452 18138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:44.721299 18146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:44.721306 18144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:44.721302 18143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:44.721382 18138 server_base.cc:1034] running on GCE node
I20260929 01:00:44.721863 18138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:44.722541 18138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:44.723729 18138 hybrid_clock.cc:648] HybridClock initialized: now 1790643644723717 us; error 28 us; skew 500 ppm
I20260929 01:00:44.725584 18138 webserver.cc:460] Webserver started at http://127.0.98.2:38419/ using document root <none> and password file <none>
I20260929 01:00:44.726159 18138 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:44.726300 18138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:44.726521 18138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:44.728721 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "d71edbeaf47e4442bce8b30547e6debb"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "922683db0e745a0687f5b372799c19fd"
server_key_iv: "e9ad1046a93907d17136110ec3d06348"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.729283 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d71edbeaf47e4442bce8b30547e6debb"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "922683db0e745a0687f5b372799c19fd"
server_key_iv: "e9ad1046a93907d17136110ec3d06348"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.732933 18138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 01:00:44.735399 18152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.736654 18138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:00:44.736801 18138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "d71edbeaf47e4442bce8b30547e6debb"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "922683db0e745a0687f5b372799c19fd"
server_key_iv: "e9ad1046a93907d17136110ec3d06348"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.736920 18138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:44.749190 18138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:44.749887 18138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:44.750092 18138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:44.750710 18138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:44.751725 18138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:44.751796 18138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.751863 18138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:44.751904 18138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.760555 18138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42267
I20260929 01:00:44.760582 18265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42267 every 8 connection(s)
I20260929 01:00:44.761456 18138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:44.765300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 18138
I20260929 01:00:44.765442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:44.765820 392 external_mini_cluster.cc:1428] Setting key b80ca9f1245e702caddf995853b633d7
I20260929 01:00:44.767911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:00:44.771737 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.763408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56127 (local address 127.0.98.62:33981)
0929 01:00:44.763579 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.763584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.764475 (+ 891us) server_negotiation.cc:407] Connection header received
0929 01:00:44.766147 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.766150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.766242 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.766330 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:44.767554 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.768148 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.769768 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.769989 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.770099 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.771243 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:44.771247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:44.771249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:44.771266 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:44.771290 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:44.771298 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:44.771380 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.771540 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.771601 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:00:44.772600 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.763682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33981 (local address 127.0.98.2:56127)
0929 01:00:44.764327 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.764367 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:44.765903 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:44.766650 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:44.766659 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:44.766773 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:44.767402 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.767422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.768276 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.768281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.769643 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.769652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.770137 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.770141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.770331 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.770966 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:44.770968 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:44.771010 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:44.771074 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:44.771387 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:44.771398 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:44.771404 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:00:44.771539 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 01:00:44.771833 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260929 01:00:44.773738 18266 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33981
I20260929 01:00:44.773988 18266 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:44.774564 18266 heartbeater.cc:511] Master 127.0.98.62:33981 requested a full tablet report, sending...
I20260929 01:00:44.775660 17947 ts_manager.cc:194] Registered new tserver with Master: d71edbeaf47e4442bce8b30547e6debb (127.0.98.2:42267)
I20260929 01:00:44.776532 17947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56127
W20260929 01:00:44.884030 18269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:44.884268 18269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:44.884305 18269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:44.888537 18269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:44.888600 18269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:44.888669 18269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:00:44.893469 18269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:44.894531 18269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:44.895833 18269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:44.905272 18275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:44.905282 18277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:44.905282 18274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:44.905370 18269 server_base.cc:1034] running on GCE node
I20260929 01:00:44.905892 18269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:44.906627 18269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:44.907830 18269 hybrid_clock.cc:648] HybridClock initialized: now 1790643644907815 us; error 29 us; skew 500 ppm
I20260929 01:00:44.909636 18269 webserver.cc:460] Webserver started at http://127.0.98.3:33527/ using document root <none> and password file <none>
I20260929 01:00:44.910287 18269 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:44.910382 18269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:44.910595 18269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:44.912735 18269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "d999aaa60d84408d98dabd95389054b5"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "843cddeaa991ff88f7a69107f11ffd74"
server_key_iv: "7b69091432b28ec100289822a180cc92"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.913309 18269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d999aaa60d84408d98dabd95389054b5"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "843cddeaa991ff88f7a69107f11ffd74"
server_key_iv: "7b69091432b28ec100289822a180cc92"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.917245 18269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:00:44.919797 18283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.920964 18269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:00:44.921097 18269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "d999aaa60d84408d98dabd95389054b5"
format_stamp: "Formatted at 2026-09-29 01:00:44 on dist-test-slave-30s6"
server_key: "843cddeaa991ff88f7a69107f11ffd74"
server_key_iv: "7b69091432b28ec100289822a180cc92"
server_key_version: "encryptionkey@0"
I20260929 01:00:44.921216 18269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:44.932680 18269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:44.933375 18269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:44.933568 18269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:44.934257 18269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:44.935181 18269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:44.935252 18269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.935316 18269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:44.935377 18269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:44.944519 18269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37025
I20260929 01:00:44.944545 18396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37025 every 8 connection(s)
I20260929 01:00:44.945479 18269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:44.954438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 18269
I20260929 01:00:44.954550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:44.954594 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.947365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58257 (local address 127.0.98.62:33981)
0929 01:00:44.947507 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.947511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.948457 (+ 946us) server_negotiation.cc:407] Connection header received
0929 01:00:44.949766 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.949768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.949815 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.949888 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:44.950803 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.951440 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.952707 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.952921 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.953000 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.953983 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:44.953986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:44.953988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:44.954000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:44.954019 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:44.954027 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:44.954085 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.954355 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.954428 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:00:44.954855 392 external_mini_cluster.cc:1428] Setting key ae16f7c083bbd5a2dd8cbb2ddb35d75e
I20260929 01:00:44.955514 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.947669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33981 (local address 127.0.98.3:58257)
0929 01:00:44.948310 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.948351 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:44.949566 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:44.950018 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:44.950024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:44.950105 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:44.950677 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.950691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.951556 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.951559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.952593 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.952605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.953043 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.953046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.953182 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.953740 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:44.953742 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:44.953782 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:44.953843 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:44.954121 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:44.954128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:44.954132 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 01:00:44.954302 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 01:00:44.954543 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20260929 01:00:44.956696 18397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33981
I20260929 01:00:44.956954 18397 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:44.957464 18397 heartbeater.cc:511] Master 127.0.98.62:33981 requested a full tablet report, sending...
I20260929 01:00:44.958461 17946 ts_manager.cc:194] Registered new tserver with Master: d999aaa60d84408d98dabd95389054b5 (127.0.98.3:37025)
I20260929 01:00:44.959038 17946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58257
I20260929 01:00:44.968801 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:44.974577 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.970749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.0.98.62:33981)
0929 01:00:44.970901 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.970907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.971073 (+ 166us) server_negotiation.cc:407] Connection header received
0929 01:00:44.971239 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.971243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.971294 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.971381 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:44.971656 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.972178 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.973028 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.973256 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.973369 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.974132 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:44.974136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:44.974139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:44.974158 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:44.974199 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:44.974210 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:44.974297 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.974383 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.974434 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:00:44.977422 17946 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:44.980528 17946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:44.994803 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.989438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.0.98.3:37025)
0929 01:00:44.989825 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.989838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.990258 (+ 420us) server_negotiation.cc:407] Connection header received
0929 01:00:44.990467 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.990473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.990636 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.990783 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:44.991201 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.992132 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.993326 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.994041 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.994154 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.994382 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.994515 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.994616 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":141,"threads_started":1}
I20260929 01:00:44.994776 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.988971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37025 (local address 127.0.0.1:45418)
0929 01:00:44.990148 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.990160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:44.990323 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:44.990965 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:44.990968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:44.990985 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:44.991071 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.991077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.992266 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.992269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.993192 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.993202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.994204 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.994210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.994321 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.994360 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:00:44.994475 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 01:00:44.994597 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260929 01:00:44.995312 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.988626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39235 (local address 127.0.0.1:45984)
0929 01:00:44.989507 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.989556 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:44.989679 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:44.990312 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:44.990318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:44.990341 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:44.990481 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.990488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.991702 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.991705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.993455 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.993466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.994721 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.994725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.994820 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.994852 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:00:44.995091 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 01:00:44.995176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":261,"thread_start_us":83,"threads_started":1}
I20260929 01:00:44.995486 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.989438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.0.98.2:42267)
0929 01:00:44.990012 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.990026 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.990052 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:00:44.990128 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.990133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.990361 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.990512 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:44.990719 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.991602 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.993598 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.994141 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.994263 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.994461 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.994844 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.995215 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":92,"threads_started":1}
I20260929 01:00:44.995813 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.988759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.0.98.1:39235)
0929 01:00:44.989166 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.989182 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.989826 (+ 644us) server_negotiation.cc:407] Connection header received
0929 01:00:44.989912 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.989919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.990108 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.990323 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:44.990610 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.991573 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.993581 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.994358 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.994493 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.994653 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.995483 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.995653 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":110,"threads_started":1}
I20260929 01:00:44.995779 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.988690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.0.1:34012)
0929 01:00:44.989878 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:44.989892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:44.989982 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:44.990485 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:44.990488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:44.990503 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:44.990605 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.990611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.991743 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.991746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.993459 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:44.993466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.994521 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.994524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:44.994605 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.994624 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 01:00:44.994691 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 01:00:44.994742 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"mutex_wait_us":186,"thread_start_us":113,"threads_started":1}
I20260929 01:00:44.998063 18331 tablet_service.cc:1467] Processing CreateTablet for tablet 8293107c20bb497d8e9aa8a8d122222c (DEFAULT_TABLE table=test-table [id=5a36dfc844764fef9cff067e4744540d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:44.998122 18069 tablet_service.cc:1467] Processing CreateTablet for tablet 8293107c20bb497d8e9aa8a8d122222c (DEFAULT_TABLE table=test-table [id=5a36dfc844764fef9cff067e4744540d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:44.998637 18200 tablet_service.cc:1467] Processing CreateTablet for tablet 8293107c20bb497d8e9aa8a8d122222c (DEFAULT_TABLE table=test-table [id=5a36dfc844764fef9cff067e4744540d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:44.999114 18069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8293107c20bb497d8e9aa8a8d122222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:44.999168 18331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8293107c20bb497d8e9aa8a8d122222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:44.999503 18200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8293107c20bb497d8e9aa8a8d122222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:45.005548 18414 tablet_bootstrap.cc:492] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392: Bootstrap starting.
I20260929 01:00:45.006785 18416 tablet_bootstrap.cc:492] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Bootstrap starting.
I20260929 01:00:45.008087 18414 tablet_bootstrap.cc:654] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:45.008767 18416 tablet_bootstrap.cc:654] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:45.008816 18414 log.cc:824] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:45.009445 18416 log.cc:824] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:45.010035 18415 tablet_bootstrap.cc:492] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Bootstrap starting.
I20260929 01:00:45.010768 18414 tablet_bootstrap.cc:492] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392: No bootstrap required, opened a new log
I20260929 01:00:45.010962 18414 ts_tablet_manager.cc:1397] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:45.011884 18415 tablet_bootstrap.cc:654] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:45.012585 18415 log.cc:824] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:45.014279 18416 tablet_bootstrap.cc:492] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: No bootstrap required, opened a new log
I20260929 01:00:45.014493 18416 ts_tablet_manager.cc:1397] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 01:00:45.014701 18414 raft_consensus.cc:348] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.014961 18414 raft_consensus.cc:374] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:45.015014 18414 raft_consensus.cc:729] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec67310f79745bc8180434635e21392, State: Initialized, Role: FOLLOWER
I20260929 01:00:45.015466 18414 consensus_queue.cc:260] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.015610 18415 tablet_bootstrap.cc:492] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: No bootstrap required, opened a new log
I20260929 01:00:45.015789 18415 ts_tablet_manager.cc:1397] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 01:00:45.016377 18414 ts_tablet_manager.cc:1428] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 01:00:45.016759 18135 heartbeater.cc:503] Master 127.0.98.62:33981 was elected leader, sending a full tablet report...
I20260929 01:00:45.018899 18416 raft_consensus.cc:348] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.019160 18416 raft_consensus.cc:374] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:45.019215 18416 raft_consensus.cc:729] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71edbeaf47e4442bce8b30547e6debb, State: Initialized, Role: FOLLOWER
I20260929 01:00:45.019721 18416 consensus_queue.cc:260] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.020002 18415 raft_consensus.cc:348] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.020237 18415 raft_consensus.cc:374] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:45.020280 18415 raft_consensus.cc:729] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d999aaa60d84408d98dabd95389054b5, State: Initialized, Role: FOLLOWER
I20260929 01:00:45.020721 18416 ts_tablet_manager.cc:1428] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:00:45.020680 18415 consensus_queue.cc:260] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.021067 18266 heartbeater.cc:503] Master 127.0.98.62:33981 was elected leader, sending a full tablet report...
I20260929 01:00:45.021319 18397 heartbeater.cc:503] Master 127.0.98.62:33981 was elected leader, sending a full tablet report...
I20260929 01:00:45.021499 18415 ts_tablet_manager.cc:1428] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 01:00:45.081655 18136 tablet.cc:2369] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:45.196544 18398 tablet.cc:2369] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:45.262804 18267 tablet.cc:2369] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:45.314972 18422 raft_consensus.cc:482] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:45.315241 18422 raft_consensus.cc:504] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.316780 18422 leader_election.cc:288] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eec67310f79745bc8180434635e21392 (127.0.98.1:39235), d71edbeaf47e4442bce8b30547e6debb (127.0.98.2:42267)
I20260929 01:00:45.320909 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:45.316997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39235 (local address 127.0.98.3:56051)
0929 01:00:45.317120 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:45.317139 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:45.317259 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:45.317588 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:45.317592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:45.317608 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:45.317755 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:45.317761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.318695 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.318700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:45.319604 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:45.319614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.320495 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.320499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:45.320606 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:45.320632 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:00:45.320711 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 01:00:45.320779 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":8}
I20260929 01:00:45.320986 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:45.317201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56051 (local address 127.0.98.1:39235)
0929 01:00:45.317328 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:45.317333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:45.317349 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:00:45.317410 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:45.317413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:45.317466 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:45.317561 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:45.317906 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.318579 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.319733 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.320367 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.320473 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:45.320538 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:45.320748 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 01:00:45.320843 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:00:45.321547 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:45.317227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55397 (local address 127.0.98.2:42267)
0929 01:00:45.317451 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:45.317456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:45.317545 (+ 89us) server_negotiation.cc:407] Connection header received
0929 01:00:45.317702 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:45.317704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:45.317743 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:45.317795 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:45.318092 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.318715 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.320068 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.320749 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.320854 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:45.320912 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:45.321293 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 01:00:45.321388 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260929 01:00:45.321646 18089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8293107c20bb497d8e9aa8a8d122222c" candidate_uuid: "d999aaa60d84408d98dabd95389054b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eec67310f79745bc8180434635e21392" is_pre_election: true
I20260929 01:00:45.321974 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:45.317089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.98.3:55397)
0929 01:00:45.317458 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:45.317470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:45.317588 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:45.317835 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:45.317840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:45.317862 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:45.317971 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:45.317976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.318836 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.318842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:45.319921 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:45.319932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.320991 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.320999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:45.321125 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:45.321157 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:00:45.321746 (+ 589us) client_negotiation.cc:240] Negotiation successful
0929 01:00:45.321839 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":27,"thread_start_us":130,"threads_started":1}
I20260929 01:00:45.321947 18089 raft_consensus.cc:2454] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d999aaa60d84408d98dabd95389054b5 in term 0.
I20260929 01:00:45.322494 18285 leader_election.cc:302] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d999aaa60d84408d98dabd95389054b5, eec67310f79745bc8180434635e21392; no voters:
I20260929 01:00:45.322799 18422 raft_consensus.cc:2781] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:45.322783 18220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8293107c20bb497d8e9aa8a8d122222c" candidate_uuid: "d999aaa60d84408d98dabd95389054b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71edbeaf47e4442bce8b30547e6debb" is_pre_election: true
I20260929 01:00:45.322893 18422 raft_consensus.cc:482] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:45.322953 18422 raft_consensus.cc:3037] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:45.323050 18220 raft_consensus.cc:2454] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d999aaa60d84408d98dabd95389054b5 in term 0.
I20260929 01:00:45.324293 18422 raft_consensus.cc:504] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.324721 18422 leader_election.cc:288] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [CANDIDATE]: Term 1 election: Requested vote from peers eec67310f79745bc8180434635e21392 (127.0.98.1:39235), d71edbeaf47e4442bce8b30547e6debb (127.0.98.2:42267)
I20260929 01:00:45.325075 18220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8293107c20bb497d8e9aa8a8d122222c" candidate_uuid: "d999aaa60d84408d98dabd95389054b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71edbeaf47e4442bce8b30547e6debb"
I20260929 01:00:45.325212 18220 raft_consensus.cc:3037] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:45.326352 18089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8293107c20bb497d8e9aa8a8d122222c" candidate_uuid: "d999aaa60d84408d98dabd95389054b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eec67310f79745bc8180434635e21392"
I20260929 01:00:45.326520 18220 raft_consensus.cc:2454] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d999aaa60d84408d98dabd95389054b5 in term 1.
I20260929 01:00:45.326509 18089 raft_consensus.cc:3037] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:45.326843 18286 leader_election.cc:302] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d71edbeaf47e4442bce8b30547e6debb, d999aaa60d84408d98dabd95389054b5; no voters:
I20260929 01:00:45.327004 18422 raft_consensus.cc:2781] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:45.327195 18422 raft_consensus.cc:686] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 1 LEADER]: Becoming Leader. State: Replica: d999aaa60d84408d98dabd95389054b5, State: Running, Role: LEADER
I20260929 01:00:45.327529 18422 consensus_queue.cc:237] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } }
I20260929 01:00:45.327904 18089 raft_consensus.cc:2454] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d999aaa60d84408d98dabd95389054b5 in term 1.
I20260929 01:00:45.331051 17947 catalog_manager.cc:5495] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d999aaa60d84408d98dabd95389054b5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d999aaa60d84408d98dabd95389054b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d999aaa60d84408d98dabd95389054b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37025 } health_report { overall_health: HEALTHY } } }
I20260929 01:00:45.360872 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:45.356928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.0.98.3:37025)
0929 01:00:45.357096 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:45.357103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:45.357120 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:00:45.357221 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:45.357223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:45.357278 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:45.357397 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:45.357668 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.358307 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.359251 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:45.359511 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:45.359636 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:45.359978 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:45.360450 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:45.360602 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:45.360648 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 01:00:45.360703 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:00:45.366961 18220 raft_consensus.cc:1261] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 1 FOLLOWER]: Refusing update from remote peer d999aaa60d84408d98dabd95389054b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:45.366961 18089 raft_consensus.cc:1261] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392 [term 1 FOLLOWER]: Refusing update from remote peer d999aaa60d84408d98dabd95389054b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:45.367769 18424 consensus_queue.cc:1035] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d71edbeaf47e4442bce8b30547e6debb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:45.367956 18422 consensus_queue.cc:1035] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec67310f79745bc8180434635e21392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:45.376581 18429 mvcc.cc:205] Tried to move back new op lower bound from 7334476371415384064 to 7334476371264073728. Current Snapshot: MvccSnapshot[applied={T|T < 7334476371415384064}]
I20260929 01:00:45.383513 17947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:45.383661 17947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:45.386045 17947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ba7d2886d1d40cfa44963ad28e75ccd: Sending DeleteTablet for 3 replicas of tablet 8293107c20bb497d8e9aa8a8d122222c
I20260929 01:00:45.386860 18069 tablet_service.cc:1514] Processing DeleteTablet for tablet 8293107c20bb497d8e9aa8a8d122222c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:45 UTC) from {username='slave'} at 127.0.0.1:45984
I20260929 01:00:45.386858 18200 tablet_service.cc:1514] Processing DeleteTablet for tablet 8293107c20bb497d8e9aa8a8d122222c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:45 UTC) from {username='slave'} at 127.0.0.1:34012
I20260929 01:00:45.386945 18331 tablet_service.cc:1514] Processing DeleteTablet for tablet 8293107c20bb497d8e9aa8a8d122222c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:45 UTC) from {username='slave'} at 127.0.0.1:45418
I20260929 01:00:45.387439 18437 tablet_replica.cc:330] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: stopping tablet replica
I20260929 01:00:45.387704 18439 tablet_replica.cc:330] T 8293107c20bb497d8e9aa8a8d122222c P eec67310f79745bc8180434635e21392: stopping tablet replica
I20260929 01:00:45.388036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 18007
I20260929 01:00:45.390407 18437 raft_consensus.cc:2229] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:45.390784 18437 raft_consensus.cc:2258] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:45.391197 18438 tablet_replica.cc:330] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: stopping tablet replica
I20260929 01:00:45.391517 18438 raft_consensus.cc:2229] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:45.391870 18438 raft_consensus.cc:2258] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:45.392683 18437 ts_tablet_manager.cc:1905] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:45.393316 18438 ts_tablet_manager.cc:1905] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 01:00:45.395586 17932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39235 (error 108)
I20260929 01:00:45.395809 18437 ts_tablet_manager.cc:1918] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:45.395987 18437 log.cc:1196] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/8293107c20bb497d8e9aa8a8d122222c
I20260929 01:00:45.396116 18438 ts_tablet_manager.cc:1918] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:45.396212 18438 log.cc:1196] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/8293107c20bb497d8e9aa8a8d122222c
I20260929 01:00:45.396505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 18138
I20260929 01:00:45.396528 18438 ts_tablet_manager.cc:1939] T 8293107c20bb497d8e9aa8a8d122222c P d71edbeaf47e4442bce8b30547e6debb: Deleting consensus metadata
I20260929 01:00:45.396528 18437 ts_tablet_manager.cc:1939] T 8293107c20bb497d8e9aa8a8d122222c P d999aaa60d84408d98dabd95389054b5: Deleting consensus metadata
I20260929 01:00:45.398324 17934 catalog_manager.cc:4841] TS d999aaa60d84408d98dabd95389054b5 (127.0.98.3:37025): tablet 8293107c20bb497d8e9aa8a8d122222c (table test-table [id=5a36dfc844764fef9cff067e4744540d]) successfully deleted
I20260929 01:00:45.403051 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:45.402746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39235 (local address 127.0.0.1:45994)
0929 01:00:45.402893 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:45.402973 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:00:45.403307 17932 catalog_manager.cc:4584] TS eec67310f79745bc8180434635e21392 (127.0.98.1:39235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8293107c20bb497d8e9aa8a8d122222c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39235: connect: Connection refused (error 111)
I20260929 01:00:45.405965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 18269
I20260929 01:00:45.406625 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:45.405946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.0.1:34028)
0929 01:00:45.406084 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:45.406138 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 01:00:45.411793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 17916
2026-09-29T01:00:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1294 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:45 dist-test-slave-30s6 krb5kdc[18443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:45 dist-test-slave-30s6 krb5kdc[18443](info): set up 2 sockets
Sep 29 01:00:45 dist-test-slave-30s6 krb5kdc[18443](info): commencing operation
krb5kdc: starting...
W20260929 01:00:46.514122 392 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:00:46 dist-test-slave-30s6 krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:00:46.634905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42389
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:46.743409 18456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:46.743752 18456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:46.743851 18456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:46.747938 18456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:46.748034 18456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:46.748075 18456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:46.748129 18456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:46.748167 18456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:46.753504 18456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:46.754617 18456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:46.755676 18456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:46.763934 18464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:46.764174 18456 server_base.cc:1034] running on GCE node
W20260929 01:00:46.763940 18462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:46.763949 18461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:46.764855 18456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:46.765918 18456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:46.767079 18456 hybrid_clock.cc:648] HybridClock initialized: now 1790643646767021 us; error 70 us; skew 500 ppm
Sep 29 01:00:46 dist-test-slave-30s6 krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:46.774636 18456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:00:46.775647 18456 webserver.cc:460] Webserver started at http://127.0.98.62:45729/ using document root <none> and password file <none>
I20260929 01:00:46.776187 18456 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:46.776247 18456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:46.776417 18456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:46.778527 18456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "d75fb27d3c41494ba6af3e15d5e4472f"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-30s6"
server_key: "7f071b631130c02dcc73bd5cb861d8cf"
server_key_iv: "a0eb8da0829bca172efdf24de7d6eeac"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.779043 18456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "d75fb27d3c41494ba6af3e15d5e4472f"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-30s6"
server_key: "7f071b631130c02dcc73bd5cb861d8cf"
server_key_iv: "a0eb8da0829bca172efdf24de7d6eeac"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.782804 18456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:00:46.786406 18471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:46.787940 18456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:00:46.788065 18456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "d75fb27d3c41494ba6af3e15d5e4472f"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-30s6"
server_key: "7f071b631130c02dcc73bd5cb861d8cf"
server_key_iv: "a0eb8da0829bca172efdf24de7d6eeac"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.788199 18456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:46.803951 18456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:46.807163 18456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:46.807363 18456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:46.814819 18456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42389
I20260929 01:00:46.814847 18523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42389 every 8 connection(s)
I20260929 01:00:46.815858 18456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:46.818619 18524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:46.820873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 18456
I20260929 01:00:46.821023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:46.821308 392 external_mini_cluster.cc:1428] Setting key 552d31493b1aea07e6599776924bf2e5
I20260929 01:00:46.825039 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f: Bootstrap starting.
I20260929 01:00:46.827656 18524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:46.828529 18524 log.cc:824] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:46.830672 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f: No bootstrap required, opened a new log
Sep 29 01:00:46 dist-test-slave-30s6 krb5kdc[18443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:46.834908 18524 raft_consensus.cc:348] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75fb27d3c41494ba6af3e15d5e4472f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42389 } }
I20260929 01:00:46.835203 18524 raft_consensus.cc:374] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:46.835263 18524 raft_consensus.cc:729] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75fb27d3c41494ba6af3e15d5e4472f, State: Initialized, Role: FOLLOWER
I20260929 01:00:46.835729 18524 consensus_queue.cc:260] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75fb27d3c41494ba6af3e15d5e4472f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42389 } }
I20260929 01:00:46.835896 18524 raft_consensus.cc:388] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:46.835965 18524 raft_consensus.cc:482] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:46.836057 18524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:46.837159 18524 raft_consensus.cc:504] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75fb27d3c41494ba6af3e15d5e4472f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42389 } }
I20260929 01:00:46.837248 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:46.822792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.0.98.62:42389)
0929 01:00:46.823218 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:46.823236 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:46.823280 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:00:46.824120 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:46.824144 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:46.824551 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:46.824926 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:46.825674 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:46.827185 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:46.828075 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:46.828752 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:46.828943 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:46.831593 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:46.831599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:46.831616 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:46.831644 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:46.834149 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:46.834606 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:46.834611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:46.834617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:46.834752 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:46.835020 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:46.835023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:46.835025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:46.835529 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:46.835720 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:46.835936 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 01:00:46.836241 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":110,"threads_started":1}
I20260929 01:00:46.837507 18524 leader_election.cc:302] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d75fb27d3c41494ba6af3e15d5e4472f; no voters:
I20260929 01:00:46.837852 18524 leader_election.cc:288] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:46.837931 18529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:46.838127 18529 raft_consensus.cc:686] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [term 1 LEADER]: Becoming Leader. State: Replica: d75fb27d3c41494ba6af3e15d5e4472f, State: Running, Role: LEADER
I20260929 01:00:46.838442 18529 consensus_queue.cc:237] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75fb27d3c41494ba6af3e15d5e4472f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42389 } }
I20260929 01:00:46.839139 18524 sys_catalog.cc:564] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:46.840067 18531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d75fb27d3c41494ba6af3e15d5e4472f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75fb27d3c41494ba6af3e15d5e4472f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42389 } } }
I20260929 01:00:46.840229 18531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:46.840553 18530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d75fb27d3c41494ba6af3e15d5e4472f. Latest consensus state: current_term: 1 leader_uuid: "d75fb27d3c41494ba6af3e15d5e4472f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75fb27d3c41494ba6af3e15d5e4472f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42389 } } }
I20260929 01:00:46.840593 18538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:46.840651 18530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:46.843853 18538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:46.849635 18538 catalog_manager.cc:1347] Generated new cluster ID: df53e46e092c4ddb9fe10c96c26ad06b
I20260929 01:00:46.849717 18538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:46.871058 18538 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:46.872336 18538 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:46.883514 18538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f: Generated new TSK 0
I20260929 01:00:46.884351 18538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:00:46.911821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42389
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:47.023203 18550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:47.023517 18550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:47.023592 18550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:47.027616 18550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:47.027714 18550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:47.027806 18550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:00:47.033041 18550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:47.034205 18550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:47.035252 18550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:47.044996 18555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.044996 18558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:47.045063 18550 server_base.cc:1034] running on GCE node
W20260929 01:00:47.045245 18556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:47.045712 18550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:47.046434 18550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:47.047609 18550 hybrid_clock.cc:648] HybridClock initialized: now 1790643647047596 us; error 25 us; skew 500 ppm
Sep 29 01:00:47 dist-test-slave-30s6 krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:47.050328 18550 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:00:47.051169 18550 webserver.cc:460] Webserver started at http://127.0.98.1:43815/ using document root <none> and password file <none>
I20260929 01:00:47.051731 18550 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:47.051820 18550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:47.052062 18550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:47.054172 18550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "2256c4328e554581be4f68593a5df262"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "061c50e97a29e652004fd476e9291f10"
server_key_iv: "8847fbe2587c352a24a49e818d68259a"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.054775 18550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2256c4328e554581be4f68593a5df262"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "061c50e97a29e652004fd476e9291f10"
server_key_iv: "8847fbe2587c352a24a49e818d68259a"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.058490 18550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:47.060940 18565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.062166 18550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:47.062353 18550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "2256c4328e554581be4f68593a5df262"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "061c50e97a29e652004fd476e9291f10"
server_key_iv: "8847fbe2587c352a24a49e818d68259a"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.062475 18550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:47.078639 18550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:47.081664 18550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:47.081876 18550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:47.082542 18550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:47.083374 18550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:47.083443 18550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.083510 18550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:47.083560 18550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.092485 18550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45833
I20260929 01:00:47.092569 18678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45833 every 8 connection(s)
I20260929 01:00:47.093480 18550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:47.098351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 18550
I20260929 01:00:47.098510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:47.098845 392 external_mini_cluster.cc:1428] Setting key 2c367ac35003cc782a65fe5cc303353a
Sep 29 01:00:47 dist-test-slave-30s6 krb5kdc[18443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:47.108667 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.095615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43677 (local address 127.0.98.62:42389)
0929 01:00:47.095774 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.095779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.096549 (+ 770us) server_negotiation.cc:407] Connection header received
0929 01:00:47.097368 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.097371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.097416 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.097498 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:47.098822 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.099471 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.100927 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.101168 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.101269 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.104472 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:47.104479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:47.104484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:47.104520 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:47.106417 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.106930 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.106934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.106936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.107031 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.107376 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.107381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.107385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.107632 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:47.107745 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.108290 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.108468 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:00:47.109577 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.095915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42389 (local address 127.0.98.1:43677)
0929 01:00:47.096389 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.096432 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.097181 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.097666 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.097673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.098021 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:47.098676 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.098691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.099608 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.099611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.100766 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.100779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.101338 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.101345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.101594 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.102424 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:47.102440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:47.104234 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:47.106538 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.106543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.106555 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.106810 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.107133 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.107135 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.107137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.107277 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.107763 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:47.107769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:47.108030 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.108212 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.108595 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260929 01:00:47.111104 18679 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42389
I20260929 01:00:47.111377 18679 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:00:47.111992 18679 heartbeater.cc:511] Master 127.0.98.62:42389 requested a full tablet report, sending...
I20260929 01:00:47.113643 18484 ts_manager.cc:194] Registered new tserver with Master: 2256c4328e554581be4f68593a5df262 (127.0.98.1:45833)
I20260929 01:00:47.114871 18484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:00:47.129066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42389
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:47.238701 18684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:47.239008 18684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:47.239055 18684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:47.243310 18684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:47.243480 18684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:47.243572 18684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:00:47.248842 18684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:47.249825 18684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:47.250854 18684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:47.260454 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.260454 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.260457 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:47.260775 18684 server_base.cc:1034] running on GCE node
I20260929 01:00:47.261304 18684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:47.262008 18684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:47.263180 18684 hybrid_clock.cc:648] HybridClock initialized: now 1790643647263163 us; error 36 us; skew 500 ppm
Sep 29 01:00:47 dist-test-slave-30s6 krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:47.266340 18684 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:00:47.267272 18684 webserver.cc:460] Webserver started at http://127.0.98.2:33839/ using document root <none> and password file <none>
I20260929 01:00:47.267836 18684 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:47.267896 18684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:47.268107 18684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:47.270123 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "7d2d4c3fc77b48d2a85d371600af4c7b"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "fd731e8c5c3ba2403ce8cc02aaa6f6e1"
server_key_iv: "bda5f22ae4b4548ea524a96fade63c33"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.270751 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7d2d4c3fc77b48d2a85d371600af4c7b"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "fd731e8c5c3ba2403ce8cc02aaa6f6e1"
server_key_iv: "bda5f22ae4b4548ea524a96fade63c33"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.274389 18684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:47.276840 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.278048 18684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:47.278242 18684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "7d2d4c3fc77b48d2a85d371600af4c7b"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "fd731e8c5c3ba2403ce8cc02aaa6f6e1"
server_key_iv: "bda5f22ae4b4548ea524a96fade63c33"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.278371 18684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:47.309545 18684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:47.313016 18684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:47.313218 18684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:47.313848 18684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:47.314702 18684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:47.314774 18684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.314843 18684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:47.314886 18684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.324162 18684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46811
I20260929 01:00:47.324199 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46811 every 8 connection(s)
I20260929 01:00:47.325106 18684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:47.325503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 18684
I20260929 01:00:47.325659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:47.326020 392 external_mini_cluster.cc:1428] Setting key d75934a67611886a16c2e628808cdccb
Sep 29 01:00:47 dist-test-slave-30s6 krb5kdc[18443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:00:47.341583 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.327309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43229 (local address 127.0.98.62:42389)
0929 01:00:47.327452 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.327456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.328602 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 01:00:47.329677 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.329680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.329739 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.329870 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:47.331446 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.331975 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.334099 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.334372 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.334458 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.337521 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:47.337526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:47.337531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:47.337572 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:47.339164 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.339692 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.339699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.339704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.339798 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.340233 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.340236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.340238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.340475 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:47.340605 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.341294 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.341439 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:00:47.342574 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.327845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42389 (local address 127.0.98.2:43229)
0929 01:00:47.328433 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.328481 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.329406 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.330065 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.330072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.330530 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:47.331255 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.331278 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.332116 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.332124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.333956 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.333972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.334548 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.334555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.334792 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.335484 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:47.335508 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:47.337314 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:47.339290 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.339295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.339307 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.339561 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.339906 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.339912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.339914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.340109 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.340617 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:47.340627 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:47.341008 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.341231 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.341632 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":150,"threads_started":1}
I20260929 01:00:47.343804 18813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42389
I20260929 01:00:47.344045 18813 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:47.344587 18813 heartbeater.cc:511] Master 127.0.98.62:42389 requested a full tablet report, sending...
I20260929 01:00:47.345768 18484 ts_manager.cc:194] Registered new tserver with Master: 7d2d4c3fc77b48d2a85d371600af4c7b (127.0.98.2:46811)
I20260929 01:00:47.346568 18484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:00:47.357610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42389
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:47.466480 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:47.466733 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:47.466775 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:47.470754 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:47.470813 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:47.470870 18818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:00:47.475961 18818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:47.476950 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:47.477906 18818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:47.487540 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.487546 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.487546 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:47.487582 18818 server_base.cc:1034] running on GCE node
I20260929 01:00:47.488262 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:47.488870 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:47.490061 18818 hybrid_clock.cc:648] HybridClock initialized: now 1790643647490049 us; error 37 us; skew 500 ppm
Sep 29 01:00:47 dist-test-slave-30s6 krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:47.492869 18818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:00:47.493782 18818 webserver.cc:460] Webserver started at http://127.0.98.3:32867/ using document root <none> and password file <none>
I20260929 01:00:47.494400 18818 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:47.494499 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:47.494724 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:47.496762 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "d3d5834c21ea47c498f5eed451c712f1"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "93f8ab9d64631ed1b7d963b66eeabaf0"
server_key_iv: "25db0e5ddb10bd63541c1b222f70d747"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.497316 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d3d5834c21ea47c498f5eed451c712f1"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "93f8ab9d64631ed1b7d963b66eeabaf0"
server_key_iv: "25db0e5ddb10bd63541c1b222f70d747"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.501158 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 01:00:47.503618 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.504810 18818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:00:47.504945 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "d3d5834c21ea47c498f5eed451c712f1"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-30s6"
server_key: "93f8ab9d64631ed1b7d963b66eeabaf0"
server_key_iv: "25db0e5ddb10bd63541c1b222f70d747"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.505054 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:47.548074 18818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:47.551497 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:47.551704 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:47.552315 18818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:47.553165 18818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:47.553239 18818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.553308 18818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:47.553363 18818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.562261 18818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33231
I20260929 01:00:47.562285 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33231 every 8 connection(s)
I20260929 01:00:47.563167 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:47.564078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 18818
I20260929 01:00:47.564208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:47.564538 392 external_mini_cluster.cc:1428] Setting key b9d281b74e4934fb9df3499c44c090da
Sep 29 01:00:47 dist-test-slave-30s6 krb5kdc[18443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:47.576629 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.565106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55255 (local address 127.0.98.62:42389)
0929 01:00:47.565238 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.565243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.566222 (+ 979us) server_negotiation.cc:407] Connection header received
0929 01:00:47.567502 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.567504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.567547 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.567613 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:47.568678 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.569205 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.570420 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.570628 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.570708 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.573122 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:47.573132 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:47.573135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:47.573158 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:47.574645 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.575165 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.575170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.575172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.575249 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.575586 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.575589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.575591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.575764 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:47.575871 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.576337 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.576459 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:00:47.577577 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.565487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42389 (local address 127.0.98.3:55255)
0929 01:00:47.566039 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.566112 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.567304 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.567746 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.567753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.568114 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:47.568566 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.568577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.569316 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.569318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.570303 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.570313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.570750 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.570753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.570900 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.571429 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:47.571446 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:47.572965 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:47.574775 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.574780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.574793 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.575046 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.575346 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.575349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.575351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.575487 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.575874 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:47.575883 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:47.576114 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.576299 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.576624 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":130,"threads_started":1}
I20260929 01:00:47.578640 18947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42389
I20260929 01:00:47.578881 18947 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:47.579332 18947 heartbeater.cc:511] Master 127.0.98.62:42389 requested a full tablet report, sending...
I20260929 01:00:47.580410 18484 ts_manager.cc:194] Registered new tserver with Master: d3d5834c21ea47c498f5eed451c712f1 (127.0.98.3:33231)
I20260929 01:00:47.581044 18484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55255
I20260929 01:00:47.590399 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:47.599515 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.592281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.0.98.62:42389)
0929 01:00:47.592421 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.592426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.592564 (+ 138us) server_negotiation.cc:407] Connection header received
0929 01:00:47.592648 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.592650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.592688 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.592752 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:47.593471 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.594022 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.594862 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.595091 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.595185 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.596585 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:47.596588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:47.596589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:47.596611 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:47.598034 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.598520 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.598523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.598524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.598594 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.598848 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.598850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.598851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.599027 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:47.599138 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.599309 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.599407 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:00:47.602170 18484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:47.605098 18484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:47.619448 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.614043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46811 (local address 127.0.0.1:58336)
0929 01:00:47.615158 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.615173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.615252 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.615691 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.615694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.615711 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:47.615812 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.615822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.616899 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.616902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.618086 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.618097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.618907 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.618911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.619028 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.619066 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.619172 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.619289 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":52,"thread_start_us":87,"threads_started":1}
I20260929 01:00:47.619637 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.614431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.0.98.3:33231)
0929 01:00:47.614786 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.614796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.614888 (+ 92us) server_negotiation.cc:407] Connection header received
0929 01:00:47.615026 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.615030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.615211 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.615358 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:47.615547 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.616619 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.618064 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.618903 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.619009 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.619138 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.619383 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.619483 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":84,"threads_started":1}
I20260929 01:00:47.619484 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.613382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33231 (local address 127.0.0.1:46632)
0929 01:00:47.614801 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.614815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.614914 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.615361 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.615364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.615377 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:47.615458 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.615463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.616789 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.616793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.617931 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.617940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.619146 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.619151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.619235 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.619253 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.619323 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.619377 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1359,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
I20260929 01:00:47.619915 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.613312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45833 (local address 127.0.0.1:34398)
0929 01:00:47.613942 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.613997 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.614121 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.614903 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.614909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.614932 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:47.615080 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.615088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.616589 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.616592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.618043 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.618054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.619211 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.619214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.619286 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.619336 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.619410 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.619477 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260929 01:00:47.620108 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.614232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.0.98.2:46811)
0929 01:00:47.614667 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.614679 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.615243 (+ 564us) server_negotiation.cc:407] Connection header received
0929 01:00:47.615357 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.615361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.615549 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.615703 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:47.615933 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.616754 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.618227 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.618778 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.618905 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.619066 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.619798 (+ 732us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.619921 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":148,"threads_started":1}
I20260929 01:00:47.620476 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.613440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.98.1:45833)
0929 01:00:47.614438 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.614446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.614470 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:00:47.614549 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.614555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.614756 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.614913 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:47.615224 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.616469 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.618151 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.618910 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.619009 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.619138 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.619556 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.619653 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":97,"threads_started":1}
I20260929 01:00:47.622864 18747 tablet_service.cc:1467] Processing CreateTablet for tablet fcb952ec7f6047b9ae88f5df3333c38e (DEFAULT_TABLE table=test-table [id=2fc3b23f49984f32bbc8ddaceadbfaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:47.622923 18881 tablet_service.cc:1467] Processing CreateTablet for tablet fcb952ec7f6047b9ae88f5df3333c38e (DEFAULT_TABLE table=test-table [id=2fc3b23f49984f32bbc8ddaceadbfaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:47.622920 18613 tablet_service.cc:1467] Processing CreateTablet for tablet fcb952ec7f6047b9ae88f5df3333c38e (DEFAULT_TABLE table=test-table [id=2fc3b23f49984f32bbc8ddaceadbfaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:47.623853 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb952ec7f6047b9ae88f5df3333c38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:47.623906 18881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb952ec7f6047b9ae88f5df3333c38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:47.623904 18613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb952ec7f6047b9ae88f5df3333c38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:47.629845 18964 tablet_bootstrap.cc:492] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: Bootstrap starting.
I20260929 01:00:47.632632 18964 tablet_bootstrap.cc:654] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:47.633651 18964 log.cc:824] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:47.634326 18965 tablet_bootstrap.cc:492] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262: Bootstrap starting.
I20260929 01:00:47.635761 18964 tablet_bootstrap.cc:492] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: No bootstrap required, opened a new log
I20260929 01:00:47.635951 18964 ts_tablet_manager.cc:1397] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:47.636462 18965 tablet_bootstrap.cc:654] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:47.637254 18965 log.cc:824] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:47.637926 18966 tablet_bootstrap.cc:492] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Bootstrap starting.
I20260929 01:00:47.639166 18965 tablet_bootstrap.cc:492] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262: No bootstrap required, opened a new log
I20260929 01:00:47.639367 18965 ts_tablet_manager.cc:1397] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:00:47.640025 18966 tablet_bootstrap.cc:654] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:47.639911 18964 raft_consensus.cc:348] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.640230 18964 raft_consensus.cc:374] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:47.640302 18964 raft_consensus.cc:729] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2d4c3fc77b48d2a85d371600af4c7b, State: Initialized, Role: FOLLOWER
I20260929 01:00:47.640805 18964 consensus_queue.cc:260] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.641179 18966 log.cc:824] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:47.641597 18813 heartbeater.cc:503] Master 127.0.98.62:42389 was elected leader, sending a full tablet report...
I20260929 01:00:47.641814 18964 ts_tablet_manager.cc:1428] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 01:00:47.643093 18966 tablet_bootstrap.cc:492] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: No bootstrap required, opened a new log
I20260929 01:00:47.643296 18966 ts_tablet_manager.cc:1397] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:47.644013 18965 raft_consensus.cc:348] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.644285 18965 raft_consensus.cc:374] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:47.644333 18965 raft_consensus.cc:729] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2256c4328e554581be4f68593a5df262, State: Initialized, Role: FOLLOWER
I20260929 01:00:47.644752 18965 consensus_queue.cc:260] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.645417 18679 heartbeater.cc:503] Master 127.0.98.62:42389 was elected leader, sending a full tablet report...
I20260929 01:00:47.645705 18965 ts_tablet_manager.cc:1428] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:00:47.647028 18966 raft_consensus.cc:348] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.647259 18966 raft_consensus.cc:374] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:47.647321 18966 raft_consensus.cc:729] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d5834c21ea47c498f5eed451c712f1, State: Initialized, Role: FOLLOWER
I20260929 01:00:47.647708 18966 consensus_queue.cc:260] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.648478 18966 ts_tablet_manager.cc:1428] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260929 01:00:47.648953 18947 heartbeater.cc:503] Master 127.0.98.62:42389 was elected leader, sending a full tablet report...
I20260929 01:00:47.765571 18972 raft_consensus.cc:482] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:47.765815 18972 raft_consensus.cc:504] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.766644 18969 raft_consensus.cc:482] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:47.766871 18969 raft_consensus.cc:504] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.767267 18972 leader_election.cc:288] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2256c4328e554581be4f68593a5df262 (127.0.98.1:45833), 7d2d4c3fc77b48d2a85d371600af4c7b (127.0.98.2:46811)
I20260929 01:00:47.768083 18969 leader_election.cc:288] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2256c4328e554581be4f68593a5df262 (127.0.98.1:45833), d3d5834c21ea47c498f5eed451c712f1 (127.0.98.3:33231)
I20260929 01:00:47.772439 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.767492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45833 (local address 127.0.98.3:35359)
0929 01:00:47.767737 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.767751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.767857 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.768330 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.768333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.768353 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:47.768500 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.768507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.769874 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.769877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.771126 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.771135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.772019 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.772022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.772126 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.772154 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.772202 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.772279 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 01:00:47.772527 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.768180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45833 (local address 127.0.98.2:34307)
0929 01:00:47.768265 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.768280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.768359 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.769042 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.769045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.769060 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:47.769198 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.769204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.770367 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.770370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.771152 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.771161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.772130 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.772133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.772256 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.772280 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.772323 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.772386 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":19}
I20260929 01:00:47.773211 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.767790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35359 (local address 127.0.98.1:45833)
0929 01:00:47.767962 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.767967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.767983 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:00:47.768035 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.768037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.768106 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.768846 (+ 740us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:47.768994 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.769759 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.771270 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.771897 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.772818 (+ 921us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.772906 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.772936 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.772986 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":9088}
I20260929 01:00:47.773191 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.768359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34307 (local address 127.0.98.1:45833)
0929 01:00:47.768744 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.768754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.768769 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:00:47.768838 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.768841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.768913 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.769291 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:47.769397 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.770236 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.771281 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.771991 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.772785 (+ 794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.772894 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.772930 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.772992 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":327,"spinlock_wait_cycles":30976,"thread_start_us":160,"threads_started":1}
I20260929 01:00:47.773411 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.767587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57841 (local address 127.0.98.2:46811)
0929 01:00:47.767714 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.767718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.768918 (+ 1200us) server_negotiation.cc:407] Connection header received
0929 01:00:47.769085 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.769088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.769126 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.769169 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:47.769513 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.770250 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.771641 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.772277 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.772362 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.772415 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.773230 (+ 815us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.773304 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":35}
I20260929 01:00:47.773878 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.768478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33231 (local address 127.0.98.2:48507)
0929 01:00:47.768738 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.768754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.768901 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.769235 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.769239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.769258 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:47.769359 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.769365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.770367 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.770370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.772431 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.772439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.773463 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.773466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.773580 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.773602 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.773672 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.773733 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260929 01:00:47.773901 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.768675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48507 (local address 127.0.98.3:33231)
0929 01:00:47.768769 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.768773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.768840 (+ 67us) server_negotiation.cc:407] Connection header received
0929 01:00:47.769021 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.769026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.769068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.769112 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:47.769479 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.770243 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.772574 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.773208 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.773290 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.773350 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.773698 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.773767 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
I20260929 01:00:47.773986 18632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb952ec7f6047b9ae88f5df3333c38e" candidate_uuid: "d3d5834c21ea47c498f5eed451c712f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2256c4328e554581be4f68593a5df262" is_pre_election: true
I20260929 01:00:47.773841 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb952ec7f6047b9ae88f5df3333c38e" candidate_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2256c4328e554581be4f68593a5df262" is_pre_election: true
I20260929 01:00:47.774329 18633 raft_consensus.cc:2454] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d2d4c3fc77b48d2a85d371600af4c7b in term 0.
I20260929 01:00:47.774497 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb952ec7f6047b9ae88f5df3333c38e" candidate_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d5834c21ea47c498f5eed451c712f1" is_pre_election: true
I20260929 01:00:47.774694 18901 raft_consensus.cc:2454] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d2d4c3fc77b48d2a85d371600af4c7b in term 0.
I20260929 01:00:47.774699 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.767510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46811 (local address 127.0.98.3:57841)
0929 01:00:47.768830 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.768842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.768982 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.769282 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.769285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.769298 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:47.769403 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.769409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.770381 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.770383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.771520 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.771529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.772489 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.772492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.773082 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.773110 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.773669 (+ 559us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.773736 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"mutex_wait_us":889,"thread_start_us":140,"threads_started":1}
I20260929 01:00:47.774896 18701 leader_election.cc:302] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2256c4328e554581be4f68593a5df262, 7d2d4c3fc77b48d2a85d371600af4c7b; no voters:
I20260929 01:00:47.775197 18969 raft_consensus.cc:2781] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:47.775264 18969 raft_consensus.cc:482] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:47.775292 18969 raft_consensus.cc:3037] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:47.775625 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb952ec7f6047b9ae88f5df3333c38e" candidate_uuid: "d3d5834c21ea47c498f5eed451c712f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" is_pre_election: true
I20260929 01:00:47.776386 18969 raft_consensus.cc:504] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.776643 18767 raft_consensus.cc:2379] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3d5834c21ea47c498f5eed451c712f1 in current term 1: Already voted for candidate 7d2d4c3fc77b48d2a85d371600af4c7b in this term.
I20260929 01:00:47.776737 18969 leader_election.cc:288] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [CANDIDATE]: Term 1 election: Requested vote from peers 2256c4328e554581be4f68593a5df262 (127.0.98.1:45833), d3d5834c21ea47c498f5eed451c712f1 (127.0.98.3:33231)
I20260929 01:00:47.777101 18632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb952ec7f6047b9ae88f5df3333c38e" candidate_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2256c4328e554581be4f68593a5df262"
I20260929 01:00:47.777148 18836 leader_election.cc:302] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3d5834c21ea47c498f5eed451c712f1; no voters: 2256c4328e554581be4f68593a5df262, 7d2d4c3fc77b48d2a85d371600af4c7b
I20260929 01:00:47.777231 18632 raft_consensus.cc:3037] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:47.777395 18972 raft_consensus.cc:3037] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:47.778321 18632 raft_consensus.cc:2454] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d2d4c3fc77b48d2a85d371600af4c7b in term 1.
I20260929 01:00:47.778532 18972 raft_consensus.cc:2726] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 01:00:47.778600 18701 leader_election.cc:302] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2256c4328e554581be4f68593a5df262, 7d2d4c3fc77b48d2a85d371600af4c7b; no voters:
I20260929 01:00:47.778570 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb952ec7f6047b9ae88f5df3333c38e" candidate_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d5834c21ea47c498f5eed451c712f1"
I20260929 01:00:47.778755 18969 raft_consensus.cc:2781] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:47.778959 18969 raft_consensus.cc:686] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 1 LEADER]: Becoming Leader. State: Replica: 7d2d4c3fc77b48d2a85d371600af4c7b, State: Running, Role: LEADER
I20260929 01:00:47.779278 18969 consensus_queue.cc:237] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } }
I20260929 01:00:47.779693 18901 raft_consensus.cc:2454] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d2d4c3fc77b48d2a85d371600af4c7b in term 1.
I20260929 01:00:47.782371 18487 catalog_manager.cc:5495] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d2d4c3fc77b48d2a85d371600af4c7b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d2d4c3fc77b48d2a85d371600af4c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46811 } health_report { overall_health: HEALTHY } } }
I20260929 01:00:47.811214 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.807734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.0.98.2:46811)
0929 01:00:47.807871 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.807874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.807888 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:00:47.808023 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.808025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.808078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.808163 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:47.808431 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.808924 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.809798 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.810051 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.810145 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.810438 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:47.810819 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:47.810934 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.811005 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.811033 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 01:00:47.814078 18948 tablet.cc:2369] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:47.817037 18632 raft_consensus.cc:1261] T fcb952ec7f6047b9ae88f5df3333c38e P 2256c4328e554581be4f68593a5df262 [term 1 FOLLOWER]: Refusing update from remote peer 7d2d4c3fc77b48d2a85d371600af4c7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:47.817093 18901 raft_consensus.cc:1261] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 1 FOLLOWER]: Refusing update from remote peer 7d2d4c3fc77b48d2a85d371600af4c7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:47.817701 18969 consensus_queue.cc:1035] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2256c4328e554581be4f68593a5df262" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:47.817888 18976 consensus_queue.cc:1035] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d5834c21ea47c498f5eed451c712f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:47.829257 18979 mvcc.cc:205] Tried to move back new op lower bound from 7334476381451571200 to 7334476381306834944. Current Snapshot: MvccSnapshot[applied={T|T < 7334476381451571200 or (T in {7334476381451571200})}]
I20260929 01:00:47.832129 18981 mvcc.cc:205] Tried to move back new op lower bound from 7334476381451571200 to 7334476381306834944. Current Snapshot: MvccSnapshot[applied={T|T < 7334476381451571200}]
I20260929 01:00:47.835952 18484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:47.836092 18484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:47.838402 18484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d75fb27d3c41494ba6af3e15d5e4472f: Sending DeleteTablet for 3 replicas of tablet fcb952ec7f6047b9ae88f5df3333c38e
I20260929 01:00:47.839298 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet fcb952ec7f6047b9ae88f5df3333c38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58336
I20260929 01:00:47.839331 18613 tablet_service.cc:1514] Processing DeleteTablet for tablet fcb952ec7f6047b9ae88f5df3333c38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34398
I20260929 01:00:47.839802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 18550
I20260929 01:00:47.840193 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet fcb952ec7f6047b9ae88f5df3333c38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46632
I20260929 01:00:47.840674 18990 tablet_replica.cc:330] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: stopping tablet replica
I20260929 01:00:47.840714 18988 tablet_replica.cc:330] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: stopping tablet replica
I20260929 01:00:47.841012 18990 raft_consensus.cc:2229] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:47.841042 18988 raft_consensus.cc:2229] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:47.841336 18990 raft_consensus.cc:2258] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:47.841483 18988 raft_consensus.cc:2258] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:47.843140 18990 ts_tablet_manager.cc:1905] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:47.845844 18990 ts_tablet_manager.cc:1918] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:47.845952 18990 log.cc:1196] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/fcb952ec7f6047b9ae88f5df3333c38e
I20260929 01:00:47.846285 18990 ts_tablet_manager.cc:1939] T fcb952ec7f6047b9ae88f5df3333c38e P d3d5834c21ea47c498f5eed451c712f1: Deleting consensus metadata
I20260929 01:00:47.847193 18988 ts_tablet_manager.cc:1905] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:47.847520 18474 catalog_manager.cc:4841] TS d3d5834c21ea47c498f5eed451c712f1 (127.0.98.3:33231): tablet fcb952ec7f6047b9ae88f5df3333c38e (table test-table [id=2fc3b23f49984f32bbc8ddaceadbfaa0]) successfully deleted
I20260929 01:00:47.850579 18988 ts_tablet_manager.cc:1918] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:47.850684 18988 log.cc:1196] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/fcb952ec7f6047b9ae88f5df3333c38e
I20260929 01:00:47.850986 18988 ts_tablet_manager.cc:1939] T fcb952ec7f6047b9ae88f5df3333c38e P 7d2d4c3fc77b48d2a85d371600af4c7b: Deleting consensus metadata
W20260929 01:00:47.851974 18473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45833 (error 108)
I20260929 01:00:47.852165 18474 catalog_manager.cc:4841] TS 7d2d4c3fc77b48d2a85d371600af4c7b (127.0.98.2:46811): tablet fcb952ec7f6047b9ae88f5df3333c38e (table test-table [id=2fc3b23f49984f32bbc8ddaceadbfaa0]) successfully deleted
I20260929 01:00:47.852203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 18684
I20260929 01:00:47.854475 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.854036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45833 (local address 127.0.0.1:34408)
0929 01:00:47.854284 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.854368 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 01:00:47.854656 18473 catalog_manager.cc:4584] TS 2256c4328e554581be4f68593a5df262 (127.0.98.1:45833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcb952ec7f6047b9ae88f5df3333c38e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45833: connect: Connection refused (error 111)
I20260929 01:00:47.858167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 18818
I20260929 01:00:47.863641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 18456
2026-09-29T01:00:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2452 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T01:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:47Z Disabled control of system clock
I20260929 01:00:47.893846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36719
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:00:48.001715 18995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:48.001987 18995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:48.002023 18995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:48.005901 18995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:48.005963 18995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:48.005980 18995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:48.005997 18995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:48.006012 18995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:48.010795 18995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:48.011861 18995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:48.013147 18995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:48.021659 19000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:48.021656 19003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:48.021672 19001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:48.021944 18995 server_base.cc:1034] running on GCE node
I20260929 01:00:48.022565 18995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:48.023663 18995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:48.024845 18995 hybrid_clock.cc:648] HybridClock initialized: now 1790643648024819 us; error 52 us; skew 500 ppm
I20260929 01:00:48.026759 18995 webserver.cc:460] Webserver started at http://127.0.98.62:46505/ using document root <none> and password file <none>
I20260929 01:00:48.027350 18995 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:48.027446 18995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:48.027688 18995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:48.029819 18995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "b66e922234e4438c93bca47c59c8318e"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "4bf0dabd802886c2eac96aba2c4737c6"
server_key_iv: "1b8aca710e41396b80504de4dc6ff5be"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.030448 18995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "b66e922234e4438c93bca47c59c8318e"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "4bf0dabd802886c2eac96aba2c4737c6"
server_key_iv: "1b8aca710e41396b80504de4dc6ff5be"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.034106 18995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:00:48.036564 19009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.037772 18995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:48.037912 18995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "b66e922234e4438c93bca47c59c8318e"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "4bf0dabd802886c2eac96aba2c4737c6"
server_key_iv: "1b8aca710e41396b80504de4dc6ff5be"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.038025 18995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:48.049825 18995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:48.050576 18995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:48.050767 18995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:48.057572 18995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36719
I20260929 01:00:48.057577 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36719 every 8 connection(s)
I20260929 01:00:48.058564 18995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:48.059247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 18995
I20260929 01:00:48.059391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:48.059661 392 external_mini_cluster.cc:1428] Setting key 61daf097aa02ace8c0e34090066d1dec
I20260929 01:00:48.062136 19062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:48.068919 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e: Bootstrap starting.
I20260929 01:00:48.069976 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.060944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.0.98.62:36719)
0929 01:00:48.061615 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.061634 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.061679 (+ 45us) server_negotiation.cc:407] Connection header received
0929 01:00:48.062560 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.062600 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.063089 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.063452 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.063897 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.065393 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.066319 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.066999 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.067168 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.067997 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.068002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.068019 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:48.068500 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.068580 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:48.068607 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.068706 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.068824 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.069089 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":100,"threads_started":1}
I20260929 01:00:48.071527 19062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:48.072353 19062 log.cc:824] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:48.074137 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e: No bootstrap required, opened a new log
I20260929 01:00:48.077585 19062 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e922234e4438c93bca47c59c8318e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36719 } }
I20260929 01:00:48.077852 19062 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:48.077916 19062 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66e922234e4438c93bca47c59c8318e, State: Initialized, Role: FOLLOWER
I20260929 01:00:48.078445 19062 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e922234e4438c93bca47c59c8318e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36719 } }
I20260929 01:00:48.078572 19062 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:48.078620 19062 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:48.078691 19062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:48.079535 19062 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e922234e4438c93bca47c59c8318e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36719 } }
I20260929 01:00:48.079821 19062 leader_election.cc:302] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66e922234e4438c93bca47c59c8318e; no voters:
I20260929 01:00:48.080067 19062 leader_election.cc:288] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:48.080204 19067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:48.080457 19067 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [term 1 LEADER]: Becoming Leader. State: Replica: b66e922234e4438c93bca47c59c8318e, State: Running, Role: LEADER
I20260929 01:00:48.080760 19067 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e922234e4438c93bca47c59c8318e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36719 } }
I20260929 01:00:48.081367 19062 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:48.082458 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66e922234e4438c93bca47c59c8318e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e922234e4438c93bca47c59c8318e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36719 } } }
I20260929 01:00:48.082604 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:48.082854 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66e922234e4438c93bca47c59c8318e. Latest consensus state: current_term: 1 leader_uuid: "b66e922234e4438c93bca47c59c8318e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e922234e4438c93bca47c59c8318e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36719 } } }
I20260929 01:00:48.082945 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:48.082962 19076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:48.085692 19076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:48.091264 19076 catalog_manager.cc:1347] Generated new cluster ID: d81a58e7ca264c05b94903cf2328c1ca
I20260929 01:00:48.091329 19076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:48.098958 19076 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:48.100138 19076 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:48.106432 19076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e: Generated new TSK 0
I20260929 01:00:48.107197 19076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:48.114965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36719
--builtin_ntp_servers=127.0.98.20:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:00:48.225214 19086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:48.225476 19086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:48.225514 19086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:48.229501 19086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:48.229557 19086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:48.229614 19086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:00:48.234375 19086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:48.235397 19086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:48.236688 19086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:48.246212 19091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:48.246117 19092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:48.246125 19094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:48.246476 19086 server_base.cc:1034] running on GCE node
I20260929 01:00:48.246853 19086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:48.247465 19086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:48.248618 19086 hybrid_clock.cc:648] HybridClock initialized: now 1790643648248584 us; error 49 us; skew 500 ppm
I20260929 01:00:48.250454 19086 webserver.cc:460] Webserver started at http://127.0.98.1:39763/ using document root <none> and password file <none>
I20260929 01:00:48.250983 19086 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:48.251040 19086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:48.251196 19086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:48.253126 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "538d3d4390d24f9a40070b4dd2c63694"
server_key_iv: "9010fccfdd30178c03737a2f0bcc319b"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.253631 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "538d3d4390d24f9a40070b4dd2c63694"
server_key_iv: "9010fccfdd30178c03737a2f0bcc319b"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.257149 19086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 01:00:48.259449 19100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.260679 19086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:00:48.260789 19086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "538d3d4390d24f9a40070b4dd2c63694"
server_key_iv: "9010fccfdd30178c03737a2f0bcc319b"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.260892 19086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:48.273617 19086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:48.274302 19086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:48.274482 19086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:48.275027 19086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:48.275933 19086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:48.275979 19086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.276041 19086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:48.276074 19086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.284864 19086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39855
I20260929 01:00:48.284885 19213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39855 every 8 connection(s)
I20260929 01:00:48.285804 19086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:48.291179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 19086
I20260929 01:00:48.291285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:48.291556 392 external_mini_cluster.cc:1428] Setting key 79a71769baf865b06a2d2167f8ec1cbe
I20260929 01:00:48.293391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36719
--builtin_ntp_servers=127.0.98.20:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:00:48.295125 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.287623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58331 (local address 127.0.98.62:36719)
0929 01:00:48.287814 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.287818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.288854 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 01:00:48.290315 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.290318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.290376 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.290460 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.291362 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.291922 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.293101 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.293330 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.293443 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.294656 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.294660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.294663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:48.294681 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.294700 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:48.294708 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.294767 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.294942 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.295006 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 01:00:48.296162 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.288022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36719 (local address 127.0.98.1:58331)
0929 01:00:48.288694 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.288734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:48.290057 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:48.290640 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:48.290647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:48.290730 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:48.291209 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.291220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.292061 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.292067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.292972 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.292980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.293485 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.293489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.293660 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.294327 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:48.294329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:48.294368 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:48.294434 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:48.294792 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:48.294803 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:48.294808 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 01:00:48.294939 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 01:00:48.295257 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260929 01:00:48.297343 19214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36719
I20260929 01:00:48.297623 19214 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:48.298108 19214 heartbeater.cc:511] Master 127.0.98.62:36719 requested a full tablet report, sending...
I20260929 01:00:48.299770 19026 ts_manager.cc:194] Registered new tserver with Master: d3a19f9ca2a641848cd699e3b7f7f4f2 (127.0.98.1:39855)
I20260929 01:00:48.301499 19026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58331
W20260929 01:00:48.405997 19217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:48.406276 19217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:48.406315 19217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:48.410284 19217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:48.410336 19217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:48.410389 19217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:00:48.414990 19217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:48.415980 19217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:48.417294 19217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:48.426872 19223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:48.427001 19222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:48.427157 19225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:48.427323 19217 server_base.cc:1034] running on GCE node
I20260929 01:00:48.427688 19217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:48.428259 19217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:48.429419 19217 hybrid_clock.cc:648] HybridClock initialized: now 1790643648429407 us; error 23 us; skew 500 ppm
I20260929 01:00:48.431250 19217 webserver.cc:460] Webserver started at http://127.0.98.2:37201/ using document root <none> and password file <none>
I20260929 01:00:48.431757 19217 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:48.431814 19217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:48.431973 19217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:48.433887 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "b4ad5882798742bfaf6448dfe1186d80"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "1046e769a1f6e662fbdd75c4809b3b24"
server_key_iv: "9f93f28d0e1c139de755e3f4336bda00"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.434459 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b4ad5882798742bfaf6448dfe1186d80"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "1046e769a1f6e662fbdd75c4809b3b24"
server_key_iv: "9f93f28d0e1c139de755e3f4336bda00"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.437850 19217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 01:00:48.440266 19231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.441462 19217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:48.441550 19217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "b4ad5882798742bfaf6448dfe1186d80"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "1046e769a1f6e662fbdd75c4809b3b24"
server_key_iv: "9f93f28d0e1c139de755e3f4336bda00"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.441628 19217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:48.456630 19217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:48.457237 19217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:48.457381 19217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:48.457873 19217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:48.458791 19217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:48.458834 19217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.458873 19217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:48.458889 19217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.467831 19217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38807
I20260929 01:00:48.467857 19344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38807 every 8 connection(s)
I20260929 01:00:48.468689 19217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:48.470456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 19217
I20260929 01:00:48.470605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:48.470877 392 external_mini_cluster.cc:1428] Setting key 3a6ccd438bdccc48d1f75feeaab1110e
I20260929 01:00:48.472949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36719
--builtin_ntp_servers=127.0.98.20:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:00:48.479333 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.470523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32847 (local address 127.0.98.62:36719)
0929 01:00:48.470665 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.470670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.471875 (+ 1205us) server_negotiation.cc:407] Connection header received
0929 01:00:48.473494 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.473498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.473556 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.473664 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.474849 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.475371 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.477215 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.477423 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.477501 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.478870 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.478873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.478875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:48.478886 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.478905 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:48.478913 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.478971 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.479170 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.479224 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:00:48.480633 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.470933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36719 (local address 127.0.98.2:32847)
0929 01:00:48.471709 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.471766 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:48.473278 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:48.473841 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:48.473848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:48.473939 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:48.474690 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.474710 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.475511 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.475518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.477091 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.477104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.477590 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.477597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.477842 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.478551 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:48.478555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:48.478616 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:48.478710 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:48.479001 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:48.479012 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:48.479021 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:00:48.479172 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 01:00:48.479510 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260929 01:00:48.481904 19345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36719
I20260929 01:00:48.482210 19345 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:48.482728 19345 heartbeater.cc:511] Master 127.0.98.62:36719 requested a full tablet report, sending...
I20260929 01:00:48.483785 19026 ts_manager.cc:194] Registered new tserver with Master: b4ad5882798742bfaf6448dfe1186d80 (127.0.98.2:38807)
I20260929 01:00:48.484340 19026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:32847
W20260929 01:00:48.584388 19348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:48.584643 19348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:48.584718 19348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:48.588961 19348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:48.589023 19348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:48.589123 19348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:00:48.593945 19348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:48.595070 19348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:48.596515 19348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:48.606303 19356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:48.606523 19348 server_base.cc:1034] running on GCE node
W20260929 01:00:48.606303 19353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:48.606314 19354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:48.607156 19348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:48.607870 19348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:48.609053 19348 hybrid_clock.cc:648] HybridClock initialized: now 1790643648609038 us; error 28 us; skew 500 ppm
I20260929 01:00:48.610975 19348 webserver.cc:460] Webserver started at http://127.0.98.3:45043/ using document root <none> and password file <none>
I20260929 01:00:48.611564 19348 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:48.611665 19348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:48.611878 19348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:48.613972 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "d96e174793054afdb8376dd8c08ce675"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "a3fc81cd8c0626fb2670b61284da2294"
server_key_iv: "80543fa74c97eae7b7937cc01c68806b"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.614609 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d96e174793054afdb8376dd8c08ce675"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "a3fc81cd8c0626fb2670b61284da2294"
server_key_iv: "80543fa74c97eae7b7937cc01c68806b"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.618609 19348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:00:48.621217 19362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.622534 19348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:48.622691 19348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "d96e174793054afdb8376dd8c08ce675"
format_stamp: "Formatted at 2026-09-29 01:00:48 on dist-test-slave-30s6"
server_key: "a3fc81cd8c0626fb2670b61284da2294"
server_key_iv: "80543fa74c97eae7b7937cc01c68806b"
server_key_version: "encryptionkey@0"
I20260929 01:00:48.622812 19348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:48.663619 19348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:48.664474 19348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:48.664702 19348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:48.665359 19348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:48.666415 19348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:48.666486 19348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.666560 19348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:48.666611 19348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:48.675647 19348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42077
I20260929 01:00:48.675674 19475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42077 every 8 connection(s)
I20260929 01:00:48.676544 19348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:48.679929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 19348
I20260929 01:00:48.680015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:48.680282 392 external_mini_cluster.cc:1428] Setting key 89d6abe7a62c0cd10c5a9c38aef008be
I20260929 01:00:48.686329 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.678488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39181 (local address 127.0.98.62:36719)
0929 01:00:48.678638 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.678641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.679659 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 01:00:48.681310 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.681313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.681368 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.681460 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.682656 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.683290 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.684510 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.684728 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.684802 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.685841 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.685844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.685846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:48.685857 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.685876 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:48.685884 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.685942 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.686113 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.686161 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:00:48.687191 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.678882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36719 (local address 127.0.98.3:39181)
0929 01:00:48.679535 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.679575 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:48.681083 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:48.681709 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:48.681766 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:48.681856 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:48.682534 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.682561 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.683457 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.683460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.684384 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.684393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.684850 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.684852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.684990 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.685609 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:48.685610 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:48.685649 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:48.685711 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:48.685980 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:48.685987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:48.685992 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 01:00:48.686099 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 01:00:48.686416 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260929 01:00:48.688326 19476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36719
I20260929 01:00:48.688611 19476 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:48.689131 19476 heartbeater.cc:511] Master 127.0.98.62:36719 requested a full tablet report, sending...
I20260929 01:00:48.690122 19026 ts_manager.cc:194] Registered new tserver with Master: d96e174793054afdb8376dd8c08ce675 (127.0.98.3:42077)
I20260929 01:00:48.690727 19026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39181
I20260929 01:00:48.694105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:48.699363 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.695959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.0.98.62:36719)
0929 01:00:48.696080 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.696083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.696278 (+ 195us) server_negotiation.cc:407] Connection header received
0929 01:00:48.696366 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.696368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.696402 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.696467 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.696714 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.697186 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.697941 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.698134 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.698256 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.698981 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.698984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.698986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:48.698997 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.699013 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:48.699021 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.699116 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.699238 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.699267 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:00:48.702069 19026 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:48.705078 19026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:48.718775 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.713350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.0.98.1:39855)
0929 01:00:48.714051 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.714063 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.714089 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:00:48.714206 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.714215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.714437 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.714622 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:48.715121 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.716090 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.717485 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.718056 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.718184 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.718398 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.718514 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.718609 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":109,"threads_started":1}
I20260929 01:00:48.718766 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.713229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39855 (local address 127.0.0.1:57626)
0929 01:00:48.713583 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.713635 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:48.713769 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:48.714818 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:48.714823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:48.714847 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:48.714981 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.714989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.716202 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.716209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.717282 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.717291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.718213 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.718217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.718323 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.718360 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:00:48.718483 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 01:00:48.718606 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260929 01:00:48.719561 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.713904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38807 (local address 127.0.0.1:40776)
0929 01:00:48.714499 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.714511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:48.714598 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:48.715244 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:48.715246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:48.715263 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:48.715370 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.715379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.716846 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.716850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.717870 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.717879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.719068 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.719072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.719166 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.719193 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 01:00:48.719264 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 01:00:48.719413 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":30,"thread_start_us":101,"threads_started":1}
I20260929 01:00:48.719843 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.714439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.98.2:38807)
0929 01:00:48.714826 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.714835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.714853 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:00:48.714911 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.714914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.715090 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.715262 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:48.715506 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.716718 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.717987 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.718811 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.719060 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.719252 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.719299 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.719388 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260929 01:00:48.720818 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.713620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42077 (local address 127.0.0.1:53744)
0929 01:00:48.714122 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.714135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:48.714261 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:48.715426 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:48.715429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:48.715447 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:48.715527 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.715533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.717000 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.717006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.719051 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.719061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.720452 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.720456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.720541 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.720561 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:00:48.720634 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:00:48.720698 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260929 01:00:48.721293 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.714011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.0.98.3:42077)
0929 01:00:48.714944 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.714961 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.714984 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:00:48.715057 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.715065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.715286 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.715437 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:48.715645 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.716736 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.719190 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.720099 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.720223 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.720390 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.721032 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.721139 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":195,"threads_started":1}
I20260929 01:00:48.722553 19148 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09869f4de94dd89954fc8a3d3d8365 (DEFAULT_TABLE table=test-table [id=e5729a3f18dc4b41a5717635c0dfe1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:48.722777 19279 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09869f4de94dd89954fc8a3d3d8365 (DEFAULT_TABLE table=test-table [id=e5729a3f18dc4b41a5717635c0dfe1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:48.723596 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09869f4de94dd89954fc8a3d3d8365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:48.723767 19279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09869f4de94dd89954fc8a3d3d8365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:48.723767 19410 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09869f4de94dd89954fc8a3d3d8365 (DEFAULT_TABLE table=test-table [id=e5729a3f18dc4b41a5717635c0dfe1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:48.724697 19410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09869f4de94dd89954fc8a3d3d8365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:48.730159 19493 tablet_bootstrap.cc:492] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2: Bootstrap starting.
I20260929 01:00:48.730813 19494 tablet_bootstrap.cc:492] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: Bootstrap starting.
I20260929 01:00:48.732153 19495 tablet_bootstrap.cc:492] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: Bootstrap starting.
I20260929 01:00:48.732913 19493 tablet_bootstrap.cc:654] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:48.732913 19494 tablet_bootstrap.cc:654] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:48.733803 19494 log.cc:824] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:48.733803 19493 log.cc:824] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:48.733934 19495 tablet_bootstrap.cc:654] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:48.734810 19495 log.cc:824] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:48.735996 19494 tablet_bootstrap.cc:492] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: No bootstrap required, opened a new log
I20260929 01:00:48.736195 19494 ts_tablet_manager.cc:1397] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:48.736295 19493 tablet_bootstrap.cc:492] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2: No bootstrap required, opened a new log
I20260929 01:00:48.736485 19493 ts_tablet_manager.cc:1397] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 01:00:48.736738 19495 tablet_bootstrap.cc:492] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: No bootstrap required, opened a new log
I20260929 01:00:48.736934 19495 ts_tablet_manager.cc:1397] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 01:00:48.740063 19495 raft_consensus.cc:348] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.740307 19495 raft_consensus.cc:374] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:48.740360 19495 raft_consensus.cc:729] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96e174793054afdb8376dd8c08ce675, State: Initialized, Role: FOLLOWER
I20260929 01:00:48.740612 19494 raft_consensus.cc:348] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.740607 19493 raft_consensus.cc:348] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.740862 19493 raft_consensus.cc:374] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:48.740861 19494 raft_consensus.cc:374] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:48.740792 19495 consensus_queue.cc:260] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.740918 19494 raft_consensus.cc:729] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ad5882798742bfaf6448dfe1186d80, State: Initialized, Role: FOLLOWER
I20260929 01:00:48.740927 19493 raft_consensus.cc:729] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a19f9ca2a641848cd699e3b7f7f4f2, State: Initialized, Role: FOLLOWER
I20260929 01:00:48.741356 19494 consensus_queue.cc:260] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.741356 19493 consensus_queue.cc:260] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.742138 19495 ts_tablet_manager.cc:1428] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 01:00:48.742317 19494 ts_tablet_manager.cc:1428] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 01:00:48.742676 19345 heartbeater.cc:503] Master 127.0.98.62:36719 was elected leader, sending a full tablet report...
I20260929 01:00:48.742700 19476 heartbeater.cc:503] Master 127.0.98.62:36719 was elected leader, sending a full tablet report...
I20260929 01:00:48.742789 19493 ts_tablet_manager.cc:1428] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 01:00:48.743129 19214 heartbeater.cc:503] Master 127.0.98.62:36719 was elected leader, sending a full tablet report...
W20260929 01:00:48.786970 19215 tablet.cc:2369] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:48.881953 19501 raft_consensus.cc:482] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:48.882247 19501 raft_consensus.cc:504] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.883639 19501 leader_election.cc:288] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a19f9ca2a641848cd699e3b7f7f4f2 (127.0.98.1:39855), d96e174793054afdb8376dd8c08ce675 (127.0.98.3:42077)
I20260929 01:00:48.887924 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.883834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39855 (local address 127.0.98.2:48845)
0929 01:00:48.883964 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.883980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:48.884084 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:48.884416 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:48.884419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:48.884437 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:48.884564 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.884570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.885657 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.885661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.886718 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.886728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.887493 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.887498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.887596 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.887624 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:00:48.887712 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 01:00:48.887781 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"spinlock_wait_cycles":2944}
I20260929 01:00:48.887951 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.884052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48845 (local address 127.0.98.1:39855)
0929 01:00:48.884182 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.884186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.884199 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:00:48.884259 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.884261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.884302 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.884401 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:48.884686 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.885512 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.886834 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.887342 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.887423 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.887472 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.887755 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.887821 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:00:48.888182 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.883970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42077 (local address 127.0.98.2:46597)
0929 01:00:48.884333 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.884344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:48.884491 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:48.884751 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:48.884754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:48.884769 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:48.884913 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.884920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.886040 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.886042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.887024 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:48.887033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.887775 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.887778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:48.887867 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.887888 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:00:48.887966 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 01:00:48.888037 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":112,"threads_started":1}
I20260929 01:00:48.888231 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.884020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46597 (local address 127.0.98.3:42077)
0929 01:00:48.884151 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.884156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.884429 (+ 273us) server_negotiation.cc:407] Connection header received
0929 01:00:48.884603 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.884608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.884654 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.884740 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:48.885109 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.885912 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.887142 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.887651 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.887748 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.887798 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.888004 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.888087 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:00:48.888909 19430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e09869f4de94dd89954fc8a3d3d8365" candidate_uuid: "b4ad5882798742bfaf6448dfe1186d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d96e174793054afdb8376dd8c08ce675" is_pre_election: true
I20260929 01:00:48.888927 19168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e09869f4de94dd89954fc8a3d3d8365" candidate_uuid: "b4ad5882798742bfaf6448dfe1186d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" is_pre_election: true
I20260929 01:00:48.889196 19430 raft_consensus.cc:2454] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4ad5882798742bfaf6448dfe1186d80 in term 0.
I20260929 01:00:48.889212 19168 raft_consensus.cc:2454] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4ad5882798742bfaf6448dfe1186d80 in term 0.
I20260929 01:00:48.889639 19235 leader_election.cc:302] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4ad5882798742bfaf6448dfe1186d80, d96e174793054afdb8376dd8c08ce675; no voters:
I20260929 01:00:48.889909 19501 raft_consensus.cc:2781] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:48.889991 19501 raft_consensus.cc:482] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:48.890033 19501 raft_consensus.cc:3037] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:48.891067 19501 raft_consensus.cc:504] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.891382 19501 leader_election.cc:288] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [CANDIDATE]: Term 1 election: Requested vote from peers d3a19f9ca2a641848cd699e3b7f7f4f2 (127.0.98.1:39855), d96e174793054afdb8376dd8c08ce675 (127.0.98.3:42077)
I20260929 01:00:48.891665 19168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e09869f4de94dd89954fc8a3d3d8365" candidate_uuid: "b4ad5882798742bfaf6448dfe1186d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2"
I20260929 01:00:48.891728 19430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e09869f4de94dd89954fc8a3d3d8365" candidate_uuid: "b4ad5882798742bfaf6448dfe1186d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d96e174793054afdb8376dd8c08ce675"
I20260929 01:00:48.891829 19168 raft_consensus.cc:3037] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:48.891840 19430 raft_consensus.cc:3037] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:48.892848 19430 raft_consensus.cc:2454] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4ad5882798742bfaf6448dfe1186d80 in term 1.
I20260929 01:00:48.892864 19168 raft_consensus.cc:2454] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4ad5882798742bfaf6448dfe1186d80 in term 1.
I20260929 01:00:48.893129 19235 leader_election.cc:302] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4ad5882798742bfaf6448dfe1186d80, d96e174793054afdb8376dd8c08ce675; no voters:
I20260929 01:00:48.893301 19501 raft_consensus.cc:2781] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:48.893510 19501 raft_consensus.cc:686] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 1 LEADER]: Becoming Leader. State: Replica: b4ad5882798742bfaf6448dfe1186d80, State: Running, Role: LEADER
I20260929 01:00:48.893821 19501 consensus_queue.cc:237] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } }
I20260929 01:00:48.897217 19024 catalog_manager.cc:5495] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4ad5882798742bfaf6448dfe1186d80 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b4ad5882798742bfaf6448dfe1186d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4ad5882798742bfaf6448dfe1186d80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:48.909224 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.905867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.0.98.2:38807)
0929 01:00:48.905993 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.905997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.906010 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:00:48.906094 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.906095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.906143 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.906287 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:48.906487 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.906986 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.907832 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.908063 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.908143 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.908454 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:48.908833 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:48.908940 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.909021 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.909078 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:00:48.914932 19430 raft_consensus.cc:1261] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 1 FOLLOWER]: Refusing update from remote peer b4ad5882798742bfaf6448dfe1186d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:48.914940 19168 raft_consensus.cc:1261] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 1 FOLLOWER]: Refusing update from remote peer b4ad5882798742bfaf6448dfe1186d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:48.915693 19503 consensus_queue.cc:1035] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d96e174793054afdb8376dd8c08ce675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:48.915896 19501 consensus_queue.cc:1035] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a19f9ca2a641848cd699e3b7f7f4f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:48.922662 19506 mvcc.cc:205] Tried to move back new op lower bound from 7334476385948516352 to 7334476385871810560. Current Snapshot: MvccSnapshot[applied={T|T < 7334476385948516352}]
I20260929 01:00:48.924081 19508 mvcc.cc:205] Tried to move back new op lower bound from 7334476385948516352 to 7334476385871810560. Current Snapshot: MvccSnapshot[applied={T|T < 7334476385948516352}]
I20260929 01:00:48.925400 19507 mvcc.cc:205] Tried to move back new op lower bound from 7334476385948516352 to 7334476385871810560. Current Snapshot: MvccSnapshot[applied={T|T < 7334476385948516352 or (T in {7334476385948516352})}]
I20260929 01:00:48.932166 19026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:48.932344 19026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:48.934676 19026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b66e922234e4438c93bca47c59c8318e: Sending DeleteTablet for 3 replicas of tablet 5e09869f4de94dd89954fc8a3d3d8365
I20260929 01:00:48.935470 19410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e09869f4de94dd89954fc8a3d3d8365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:48 UTC) from {username='slave'} at 127.0.0.1:53744
I20260929 01:00:48.935498 19279 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e09869f4de94dd89954fc8a3d3d8365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:48 UTC) from {username='slave'} at 127.0.0.1:40776
I20260929 01:00:48.935580 19148 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e09869f4de94dd89954fc8a3d3d8365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:48 UTC) from {username='slave'} at 127.0.0.1:57626
I20260929 01:00:48.935988 19517 tablet_replica.cc:330] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2: stopping tablet replica
I20260929 01:00:48.936281 19516 tablet_replica.cc:330] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: stopping tablet replica
I20260929 01:00:48.936306 19517 raft_consensus.cc:2229] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:48.936617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 19086
I20260929 01:00:48.936648 19517 raft_consensus.cc:2258] T 5e09869f4de94dd89954fc8a3d3d8365 P d3a19f9ca2a641848cd699e3b7f7f4f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:48.937474 19516 raft_consensus.cc:2229] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:48.937924 19518 tablet_replica.cc:330] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: stopping tablet replica
I20260929 01:00:48.938205 19518 raft_consensus.cc:2229] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:48.938421 19516 raft_consensus.cc:2258] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:48.938467 19518 raft_consensus.cc:2258] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:48.939885 19518 ts_tablet_manager.cc:1905] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:48.939900 19516 ts_tablet_manager.cc:1905] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:48.942682 19518 ts_tablet_manager.cc:1918] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:48.942682 19516 ts_tablet_manager.cc:1918] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:48.942782 19518 log.cc:1196] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/5e09869f4de94dd89954fc8a3d3d8365
I20260929 01:00:48.942937 19516 log.cc:1196] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/5e09869f4de94dd89954fc8a3d3d8365
I20260929 01:00:48.943102 19518 ts_tablet_manager.cc:1939] T 5e09869f4de94dd89954fc8a3d3d8365 P d96e174793054afdb8376dd8c08ce675: Deleting consensus metadata
I20260929 01:00:48.943420 19516 ts_tablet_manager.cc:1939] T 5e09869f4de94dd89954fc8a3d3d8365 P b4ad5882798742bfaf6448dfe1186d80: Deleting consensus metadata
I20260929 01:00:48.944370 19010 catalog_manager.cc:4841] TS b4ad5882798742bfaf6448dfe1186d80 (127.0.98.2:38807): tablet 5e09869f4de94dd89954fc8a3d3d8365 (table test-table [id=e5729a3f18dc4b41a5717635c0dfe1a0]) successfully deleted
I20260929 01:00:48.944361 19013 catalog_manager.cc:4841] TS d96e174793054afdb8376dd8c08ce675 (127.0.98.3:42077): tablet 5e09869f4de94dd89954fc8a3d3d8365 (table test-table [id=e5729a3f18dc4b41a5717635c0dfe1a0]) successfully deleted
W20260929 01:00:48.946172 19010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39855 (error 108)
I20260929 01:00:48.946349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 19217
I20260929 01:00:48.947707 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.947352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39855 (local address 127.0.0.1:57642)
0929 01:00:48.947516 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:48.947604 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 01:00:48.947953 19010 catalog_manager.cc:4584] TS d3a19f9ca2a641848cd699e3b7f7f4f2 (127.0.98.1:39855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e09869f4de94dd89954fc8a3d3d8365: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39855: connect: Connection refused (error 111)
I20260929 01:00:48.952579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 19348
I20260929 01:00:48.957935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 18995
2026-09-29T01:00:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:49 dist-test-slave-30s6 krb5kdc[19522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:49 dist-test-slave-30s6 krb5kdc[19522](info): set up 2 sockets
Sep 29 01:00:49 dist-test-slave-30s6 krb5kdc[19522](info): commencing operation
krb5kdc: starting...
W20260929 01:00:49.994115 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:00:50.115367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36191
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:50.223081 19535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:50.223387 19535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:50.223464 19535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:50.227445 19535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:50.227538 19535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:50.227577 19535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:50.227639 19535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:50.227690 19535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:50.233009 19535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:50.234076 19535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:50.235132 19535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:50.243495 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.243528 19543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.243564 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:50.243871 19535 server_base.cc:1034] running on GCE node
I20260929 01:00:50.244469 19535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:50.245553 19535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:50.246742 19535 hybrid_clock.cc:648] HybridClock initialized: now 1790643650246717 us; error 48 us; skew 500 ppm
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:50.249428 19535 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:00:50.250366 19535 webserver.cc:460] Webserver started at http://127.0.98.62:41503/ using document root <none> and password file <none>
I20260929 01:00:50.250946 19535 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:50.251039 19535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:50.251272 19535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:50.253315 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "f3a9199910fc40c592516bd0317b1140"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "3d25d9a36d436d9f2803ea29bf59520a"
server_key_iv: "24c6b0bcacb2a3d4dbdd5eb072c5be78"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.253880 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3a9199910fc40c592516bd0317b1140"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "3d25d9a36d436d9f2803ea29bf59520a"
server_key_iv: "24c6b0bcacb2a3d4dbdd5eb072c5be78"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.257607 19535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:00:50.260087 19550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.261332 19535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:50.261471 19535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "f3a9199910fc40c592516bd0317b1140"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "3d25d9a36d436d9f2803ea29bf59520a"
server_key_iv: "24c6b0bcacb2a3d4dbdd5eb072c5be78"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.261597 19535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:50.270589 19535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:50.273603 19535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:50.273818 19535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:50.281255 19535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36191
I20260929 01:00:50.281277 19602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36191 every 8 connection(s)
I20260929 01:00:50.282260 19535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:50.285045 19603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:50.290618 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140: Bootstrap starting.
I20260929 01:00:50.291455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 19535
I20260929 01:00:50.291563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:50.291873 392 external_mini_cluster.cc:1428] Setting key 170ff389476947b50229c00395737820
I20260929 01:00:50.293527 19603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:50.294440 19603 log.cc:824] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:50.296587 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140: No bootstrap required, opened a new log
I20260929 01:00:50.300568 19603 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a9199910fc40c592516bd0317b1140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36191 } }
I20260929 01:00:50.300778 19603 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:50.300839 19603 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a9199910fc40c592516bd0317b1140, State: Initialized, Role: FOLLOWER
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:50.301327 19603 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a9199910fc40c592516bd0317b1140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36191 } }
I20260929 01:00:50.301463 19603 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:50.301558 19603 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:50.301661 19603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:50.302829 19603 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a9199910fc40c592516bd0317b1140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36191 } }
I20260929 01:00:50.303169 19603 leader_election.cc:302] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3a9199910fc40c592516bd0317b1140; no voters:
I20260929 01:00:50.303540 19603 leader_election.cc:288] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:50.303613 19608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:50.303821 19608 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [term 1 LEADER]: Becoming Leader. State: Replica: f3a9199910fc40c592516bd0317b1140, State: Running, Role: LEADER
I20260929 01:00:50.304119 19608 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a9199910fc40c592516bd0317b1140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36191 } }
I20260929 01:00:50.304843 19603 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:50.305282 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3a9199910fc40c592516bd0317b1140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a9199910fc40c592516bd0317b1140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36191 } } }
I20260929 01:00:50.305387 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:50.305724 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3a9199910fc40c592516bd0317b1140. Latest consensus state: current_term: 1 leader_uuid: "f3a9199910fc40c592516bd0317b1140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a9199910fc40c592516bd0317b1140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36191 } } }
I20260929 01:00:50.305826 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:50.307772 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.293059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.0.98.62:36191)
0929 01:00:50.293574 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:50.293596 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:50.293639 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:00:50.294541 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:50.294564 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:50.294977 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:50.295366 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:50.296046 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.297502 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.298382 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.299118 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.299300 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.301884 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:50.301895 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:50.301908 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:50.301935 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:50.303799 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.304357 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.304361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.304365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.304472 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.304762 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.304767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.304769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.305291 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:50.306110 (+ 819us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:50.306305 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 01:00:50.306602 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":125,"threads_started":1}
W20260929 01:00:50.309784 19623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:00:50.309868 19623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:00:50.309974 19612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:50.310704 19612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:50.316179 19612 catalog_manager.cc:1347] Generated new cluster ID: 8d4910d092174cb2829ef043bcd552ae
I20260929 01:00:50.316246 19612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:50.326514 19612 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:50.327795 19612 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:50.335215 19612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140: Generated new TSK 0
I20260929 01:00:50.336124 19612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:00:50.365581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36191
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:50.477543 19629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:50.477810 19629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:50.477852 19629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:50.481935 19629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:50.481995 19629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:50.482101 19629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:00:50.487423 19629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:50.488498 19629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:50.489516 19629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:50.499048 19634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.499058 19637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.499019 19635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:50.499294 19629 server_base.cc:1034] running on GCE node
I20260929 01:00:50.499804 19629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:50.500452 19629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:50.501616 19629 hybrid_clock.cc:648] HybridClock initialized: now 1790643650501565 us; error 64 us; skew 500 ppm
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:50.504487 19629 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:00:50.505400 19629 webserver.cc:460] Webserver started at http://127.0.98.1:34063/ using document root <none> and password file <none>
I20260929 01:00:50.505986 19629 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:50.506098 19629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:50.506407 19629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:50.508496 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "c961b52c282947568f786b5e5f578bc5"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "c90734d9b3a7d94fa8f81d9d2c07912f"
server_key_iv: "141ef42f0c713847529062f10701df98"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.509073 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c961b52c282947568f786b5e5f578bc5"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "c90734d9b3a7d94fa8f81d9d2c07912f"
server_key_iv: "141ef42f0c713847529062f10701df98"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.512979 19629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 01:00:50.515519 19644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.516748 19629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:00:50.516883 19629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "c961b52c282947568f786b5e5f578bc5"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "c90734d9b3a7d94fa8f81d9d2c07912f"
server_key_iv: "141ef42f0c713847529062f10701df98"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.517002 19629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:50.536448 19629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:50.539479 19629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:50.539690 19629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:50.540298 19629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:50.541128 19629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:50.541208 19629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.541281 19629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:50.541332 19629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.550310 19629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33605
I20260929 01:00:50.550364 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33605 every 8 connection(s)
I20260929 01:00:50.551304 19629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:50.552071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 19629
I20260929 01:00:50.552225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:50.552520 392 external_mini_cluster.cc:1428] Setting key e32d1ef3998df36582d237b7062dbb05
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:00:50.567698 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.553427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55843 (local address 127.0.98.62:36191)
0929 01:00:50.553578 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:50.553582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:50.554768 (+ 1186us) server_negotiation.cc:407] Connection header received
0929 01:00:50.555891 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:50.555898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:50.555967 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:50.556067 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:50.557854 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.558796 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.560104 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.560584 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.560714 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.563473 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:50.563476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:50.563478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:50.563507 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:50.565207 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.565917 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.565921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.565923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.566024 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.566452 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.566459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.566464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.566744 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:50.566855 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:50.567375 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 01:00:50.567507 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:00:50.568571 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.553938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36191 (local address 127.0.98.1:55843)
0929 01:00:50.554593 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:50.554652 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:50.555651 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:50.556368 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:50.556380 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:50.556977 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:50.557691 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:50.557707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.558967 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.558973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:50.559959 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:50.559969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.560755 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.560758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:50.560916 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.561631 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:50.561648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:50.563297 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:50.565325 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:50.565330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:50.565343 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:50.565804 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:50.566163 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:50.566166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:50.566168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:50.566348 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:50.566851 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:50.566857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:50.567106 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 01:00:50.567310 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 01:00:50.567658 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":213,"threads_started":1}
I20260929 01:00:50.569895 19758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36191
I20260929 01:00:50.570164 19758 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:50.570730 19758 heartbeater.cc:511] Master 127.0.98.62:36191 requested a full tablet report, sending...
I20260929 01:00:50.572288 19567 ts_manager.cc:194] Registered new tserver with Master: c961b52c282947568f786b5e5f578bc5 (127.0.98.1:33605)
I20260929 01:00:50.573410 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:00:50.581537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36191
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:50.691598 19763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:50.691921 19763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:50.692001 19763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:50.696295 19763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:50.696386 19763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:50.696475 19763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:00:50.702468 19763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:50.703557 19763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:50.704612 19763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:00:50.714463 19763 server_base.cc:1034] running on GCE node
W20260929 01:00:50.714375 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.714392 19769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.714375 19771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:50.715124 19763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:50.715775 19763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:50.716961 19763 hybrid_clock.cc:648] HybridClock initialized: now 1790643650716947 us; error 26 us; skew 500 ppm
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:50.719722 19763 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:00:50.720683 19763 webserver.cc:460] Webserver started at http://127.0.98.2:42007/ using document root <none> and password file <none>
I20260929 01:00:50.721282 19763 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:50.721371 19763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:50.721580 19763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:50.723697 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0be499665a64ac38cd4f9b43aa254b7"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "f763e70d08878063b5bddc4ed2852350"
server_key_iv: "c1216d16efc711b0fe0ec7eaf03dc993"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.724354 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0be499665a64ac38cd4f9b43aa254b7"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "f763e70d08878063b5bddc4ed2852350"
server_key_iv: "c1216d16efc711b0fe0ec7eaf03dc993"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.728149 19763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:50.730703 19778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.731889 19763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:50.732018 19763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "a0be499665a64ac38cd4f9b43aa254b7"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "f763e70d08878063b5bddc4ed2852350"
server_key_iv: "c1216d16efc711b0fe0ec7eaf03dc993"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.732148 19763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:50.758574 19763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:50.761518 19763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:50.761731 19763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:50.762351 19763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:50.763173 19763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:50.763253 19763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.763324 19763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:50.763376 19763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.772611 19763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37717
I20260929 01:00:50.772643 19891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37717 every 8 connection(s)
I20260929 01:00:50.773509 19763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:50.778004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 19763
I20260929 01:00:50.778151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:50.778503 392 external_mini_cluster.cc:1428] Setting key dd49cd2722adaa499f97f664f8af097a
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:00:50.790518 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.775380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47741 (local address 127.0.98.62:36191)
0929 01:00:50.775581 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:50.775584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:50.776506 (+ 922us) server_negotiation.cc:407] Connection header received
0929 01:00:50.777587 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:50.777589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:50.777634 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:50.777784 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:50.779724 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.780396 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.781584 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.781944 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.782071 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.786208 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:50.786214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:50.786221 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:50.786284 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:50.788201 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.788801 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.788805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.788807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.788905 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.789244 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.789247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.789249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.789509 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:50.789628 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:50.790241 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 01:00:50.790378 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:00:50.791673 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.775798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36191 (local address 127.0.98.2:47741)
0929 01:00:50.776310 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:50.776381 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:50.777366 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:50.777973 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:50.777983 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:50.778536 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:50.779133 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:50.779147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.780535 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.780539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:50.781444 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:50.781453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.782152 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.782159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:50.782456 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.783161 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:50.783193 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:50.785871 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:50.788369 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:50.788386 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:50.788403 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:50.788691 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:50.789006 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:50.789008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:50.789009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:50.789140 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:50.789663 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:50.789670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:50.789969 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 01:00:50.790140 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 01:00:50.790488 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:00:50.792886 19892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36191
I20260929 01:00:50.793150 19892 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:50.793650 19892 heartbeater.cc:511] Master 127.0.98.62:36191 requested a full tablet report, sending...
I20260929 01:00:50.794745 19567 ts_manager.cc:194] Registered new tserver with Master: a0be499665a64ac38cd4f9b43aa254b7 (127.0.98.2:37717)
I20260929 01:00:50.795336 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:00:50.809482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36191
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:50.919715 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:50.919972 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:50.920017 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:50.924188 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:50.924281 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:50.924377 19897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:00:50.929754 19897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:50.930773 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:50.931742 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:50.941196 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.941237 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.941228 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:50.941406 19897 server_base.cc:1034] running on GCE node
I20260929 01:00:50.941927 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:50.942641 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:50.943833 19897 hybrid_clock.cc:648] HybridClock initialized: now 1790643650943806 us; error 39 us; skew 500 ppm
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:50.947157 19897 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:00:50.947935 19897 webserver.cc:460] Webserver started at http://127.0.98.3:42427/ using document root <none> and password file <none>
I20260929 01:00:50.948429 19897 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:50.948482 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:50.948652 19897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:50.950646 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "97df8e9777614e58aa903a7f2504812f"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "82d313f42e9df81ba3f59ba416c2d4c9"
server_key_iv: "771ce0eba8db90695fc3c4dfe2d04213"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.951160 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97df8e9777614e58aa903a7f2504812f"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "82d313f42e9df81ba3f59ba416c2d4c9"
server_key_iv: "771ce0eba8db90695fc3c4dfe2d04213"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.954736 19897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:50.957062 19912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.958351 19897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:50.958470 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "97df8e9777614e58aa903a7f2504812f"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-30s6"
server_key: "82d313f42e9df81ba3f59ba416c2d4c9"
server_key_iv: "771ce0eba8db90695fc3c4dfe2d04213"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.958606 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:50.971019 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:50.974385 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:50.974587 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:50.975160 19897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:50.975962 19897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:50.976006 19897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.976074 19897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:50.976107 19897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.984297 19897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38655
I20260929 01:00:50.984323 20025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38655 every 8 connection(s)
I20260929 01:00:50.985191 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:50.985613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 19897
I20260929 01:00:50.985771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:50.986133 392 external_mini_cluster.cc:1428] Setting key a8f939de04b7d23189dfb18e3ce8fee3
Sep 29 01:00:50 dist-test-slave-30s6 krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:00:50.998536 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.987059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50171 (local address 127.0.98.62:36191)
0929 01:00:50.987179 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:50.987184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:50.988077 (+ 893us) server_negotiation.cc:407] Connection header received
0929 01:00:50.989117 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:50.989121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:50.989178 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:50.989263 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:50.990577 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.991102 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.992232 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.992455 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.992573 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.994947 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:50.994955 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:50.994957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:50.994981 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:50.996485 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.997014 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.997019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.997021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.997103 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.997438 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.997443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.997445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.997633 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:50.997738 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:50.998249 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 01:00:50.998366 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:00:50.999325 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.987491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36191 (local address 127.0.98.3:50171)
0929 01:00:50.987931 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:50.987971 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:50.988874 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:50.989448 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:50.989454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:50.989899 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:50.990436 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:50.990450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.991233 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.991236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:50.992102 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:50.992111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.992598 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.992602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:50.992776 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.993349 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:50.993364 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:50.994777 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:50.996626 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:50.996631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:50.996645 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:50.996904 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:50.997203 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:50.997206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:50.997208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:50.997342 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:50.997735 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:50.997741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:50.997994 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 01:00:50.998187 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 01:00:50.998515 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260929 01:00:51.000461 20026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36191
I20260929 01:00:51.000739 20026 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:51.001329 20026 heartbeater.cc:511] Master 127.0.98.62:36191 requested a full tablet report, sending...
I20260929 01:00:51.002434 19567 ts_manager.cc:194] Registered new tserver with Master: 97df8e9777614e58aa903a7f2504812f (127.0.98.3:38655)
I20260929 01:00:51.003016 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50171
I20260929 01:00:51.013638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:51.023098 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.015704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.0.98.62:36191)
0929 01:00:51.015830 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.015833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.015989 (+ 156us) server_negotiation.cc:407] Connection header received
0929 01:00:51.016086 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.016088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.016129 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.016196 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:51.016819 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.017322 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.018117 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.018392 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.018473 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.019853 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:51.019856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:51.019858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:51.019875 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:51.021325 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:51.021753 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:51.021756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:51.021758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:51.021840 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:51.022109 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:51.022113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:51.022115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:51.022442 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:51.022566 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.022812 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.022926 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:00:51.025802 19567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:51.028730 19567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:51.043112 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.037733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33605 (local address 127.0.0.1:43894)
0929 01:00:51.038215 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.038229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:51.038304 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:51.038878 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:51.038883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:51.038905 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:51.039152 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.039159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.040196 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.040208 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.041508 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.041517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.042564 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.042568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.042680 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.042724 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 01:00:51.042830 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 01:00:51.042951 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":79,"spinlock_wait_cycles":1280,"thread_start_us":97,"threads_started":1}
I20260929 01:00:51.043275 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.036913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38655 (local address 127.0.0.1:47784)
0929 01:00:51.037379 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.037432 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:51.037559 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:51.038977 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:51.038980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:51.038992 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:51.039191 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.039200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.040632 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.040635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.041767 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.041777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.042857 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.042861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.042948 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.042969 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:00:51.043053 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 01:00:51.043127 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":109,"thread_start_us":88,"threads_started":1}
I20260929 01:00:51.043321 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.037814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.0.98.3:38655)
0929 01:00:51.038541 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.038554 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.038574 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:00:51.038649 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.038656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.038839 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.039056 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:51.039330 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.040479 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.041894 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.042739 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.042846 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.042980 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.043087 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.043179 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":99,"threads_started":1}
I20260929 01:00:51.043752 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.038049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.0.98.1:33605)
0929 01:00:51.038453 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.038468 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.038492 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:00:51.038581 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.038585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.038722 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.038878 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:51.039288 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.040047 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.041624 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.042448 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.042575 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.042726 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.043449 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.043551 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":78,"threads_started":1}
I20260929 01:00:51.044355 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.037181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.0.98.2:37717)
0929 01:00:51.037771 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.037785 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.038082 (+ 297us) server_negotiation.cc:407] Connection header received
0929 01:00:51.038354 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.038360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.038576 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.038703 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:51.039315 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.040531 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.042474 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.043301 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.043405 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.043538 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.044122 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.044215 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":147,"threads_started":1}
I20260929 01:00:51.044843 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.037027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37717 (local address 127.0.0.1:54394)
0929 01:00:51.037988 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.038002 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:51.038154 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:51.038900 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:51.038906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:51.038940 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:51.039132 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.039139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.041445 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.041450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.042354 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.042362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.043867 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.043872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.043956 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.043984 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:00:51.044600 (+ 616us) client_negotiation.cc:240] Negotiation successful
0929 01:00:51.044681 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":52,"spinlock_wait_cycles":896,"thread_start_us":85,"threads_started":1}
I20260929 01:00:51.046247 19960 tablet_service.cc:1467] Processing CreateTablet for tablet 35a3a4c931724d70a2f1591504560dd1 (DEFAULT_TABLE table=test-table [id=d5ff0a025c7b42bcb8f9f7ecd5b70c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:51.047099 19826 tablet_service.cc:1467] Processing CreateTablet for tablet 35a3a4c931724d70a2f1591504560dd1 (DEFAULT_TABLE table=test-table [id=d5ff0a025c7b42bcb8f9f7ecd5b70c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:51.047398 19960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a3a4c931724d70a2f1591504560dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:51.047947 19826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a3a4c931724d70a2f1591504560dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:51.048074 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 35a3a4c931724d70a2f1591504560dd1 (DEFAULT_TABLE table=test-table [id=d5ff0a025c7b42bcb8f9f7ecd5b70c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:51.048995 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a3a4c931724d70a2f1591504560dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:51.053900 20043 tablet_bootstrap.cc:492] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Bootstrap starting.
I20260929 01:00:51.055462 20045 tablet_bootstrap.cc:492] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5: Bootstrap starting.
I20260929 01:00:51.056087 20043 tablet_bootstrap.cc:654] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:51.056143 20044 tablet_bootstrap.cc:492] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Bootstrap starting.
I20260929 01:00:51.057080 20043 log.cc:824] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:51.057639 20045 tablet_bootstrap.cc:654] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:51.058283 20044 tablet_bootstrap.cc:654] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:51.058944 20045 log.cc:824] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:51.059397 20043 tablet_bootstrap.cc:492] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: No bootstrap required, opened a new log
I20260929 01:00:51.059439 20044 log.cc:824] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:51.059592 20043 ts_tablet_manager.cc:1397] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:51.060854 20045 tablet_bootstrap.cc:492] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5: No bootstrap required, opened a new log
I20260929 01:00:51.061028 20045 ts_tablet_manager.cc:1397] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:51.061327 20044 tablet_bootstrap.cc:492] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: No bootstrap required, opened a new log
I20260929 01:00:51.061491 20044 ts_tablet_manager.cc:1397] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 01:00:51.062965 20043 raft_consensus.cc:348] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.063313 20043 raft_consensus.cc:374] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:51.063369 20043 raft_consensus.cc:729] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0be499665a64ac38cd4f9b43aa254b7, State: Initialized, Role: FOLLOWER
I20260929 01:00:51.063819 20043 consensus_queue.cc:260] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.064432 19892 heartbeater.cc:503] Master 127.0.98.62:36191 was elected leader, sending a full tablet report...
I20260929 01:00:51.064750 20043 ts_tablet_manager.cc:1428] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:00:51.065171 20044 raft_consensus.cc:348] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.065428 20044 raft_consensus.cc:374] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:51.065485 20044 raft_consensus.cc:729] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df8e9777614e58aa903a7f2504812f, State: Initialized, Role: FOLLOWER
I20260929 01:00:51.065924 20044 consensus_queue.cc:260] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.065968 20045 raft_consensus.cc:348] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.066254 20045 raft_consensus.cc:374] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:51.066325 20045 raft_consensus.cc:729] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c961b52c282947568f786b5e5f578bc5, State: Initialized, Role: FOLLOWER
I20260929 01:00:51.066672 20026 heartbeater.cc:503] Master 127.0.98.62:36191 was elected leader, sending a full tablet report...
I20260929 01:00:51.066762 20045 consensus_queue.cc:260] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.066869 20044 ts_tablet_manager.cc:1428] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 01:00:51.067631 20045 ts_tablet_manager.cc:1428] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 01:00:51.067963 19758 heartbeater.cc:503] Master 127.0.98.62:36191 was elected leader, sending a full tablet report...
W20260929 01:00:51.236340 20027 tablet.cc:2369] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:51.274731 19893 tablet.cc:2369] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:51.302824 19759 tablet.cc:2369] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:51.323580 20051 raft_consensus.cc:482] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:51.323830 20051 raft_consensus.cc:504] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.325133 20051 leader_election.cc:288] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df8e9777614e58aa903a7f2504812f (127.0.98.3:38655), a0be499665a64ac38cd4f9b43aa254b7 (127.0.98.2:37717)
I20260929 01:00:51.329113 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.325694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47843 (local address 127.0.98.2:37717)
0929 01:00:51.325831 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.325838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.325857 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:00:51.326068 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.326071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.326117 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.326219 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:51.326488 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.326994 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.328150 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.328548 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.328672 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.328750 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.328920 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.328992 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:00:51.329135 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.325311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37717 (local address 127.0.98.1:47843)
0929 01:00:51.325795 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.325810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:51.325943 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:51.326248 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:51.326252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:51.326275 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:51.326373 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.326378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.327115 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.327120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.328020 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.328030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.328677 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.328680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.328781 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.328810 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:00:51.328902 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 01:00:51.328977 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":32,"spinlock_wait_cycles":44928,"thread_start_us":89,"threads_started":1}
I20260929 01:00:51.329172 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.325441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39597 (local address 127.0.98.3:38655)
0929 01:00:51.325567 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.325572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.325868 (+ 296us) server_negotiation.cc:407] Connection header received
0929 01:00:51.326058 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.326063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.326119 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.326222 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:51.326489 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.326993 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.328170 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.328607 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.328729 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.328801 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.328988 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.329059 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:00:51.329198 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.325311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38655 (local address 127.0.98.1:39597)
0929 01:00:51.325757 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.325770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:51.325946 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:51.326228 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:51.326231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:51.326247 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:51.326383 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.326389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.327113 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.327116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.328026 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.328035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.328734 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.328738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.328849 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.328875 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:00:51.328961 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 01:00:51.329041 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":77,"spinlock_wait_cycles":15488,"thread_start_us":116,"threads_started":1}
I20260929 01:00:51.329787 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a3a4c931724d70a2f1591504560dd1" candidate_uuid: "c961b52c282947568f786b5e5f578bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0be499665a64ac38cd4f9b43aa254b7" is_pre_election: true
I20260929 01:00:51.329805 19980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a3a4c931724d70a2f1591504560dd1" candidate_uuid: "c961b52c282947568f786b5e5f578bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df8e9777614e58aa903a7f2504812f" is_pre_election: true
I20260929 01:00:51.330097 19846 raft_consensus.cc:2454] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c961b52c282947568f786b5e5f578bc5 in term 0.
I20260929 01:00:51.330107 19980 raft_consensus.cc:2454] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c961b52c282947568f786b5e5f578bc5 in term 0.
I20260929 01:00:51.330651 19648 leader_election.cc:302] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 97df8e9777614e58aa903a7f2504812f, a0be499665a64ac38cd4f9b43aa254b7, c961b52c282947568f786b5e5f578bc5; no voters:
I20260929 01:00:51.330946 20051 raft_consensus.cc:2781] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:51.331022 20051 raft_consensus.cc:482] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:51.331053 20051 raft_consensus.cc:3037] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:51.332152 20051 raft_consensus.cc:504] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.332579 20051 leader_election.cc:288] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [CANDIDATE]: Term 1 election: Requested vote from peers 97df8e9777614e58aa903a7f2504812f (127.0.98.3:38655), a0be499665a64ac38cd4f9b43aa254b7 (127.0.98.2:37717)
I20260929 01:00:51.332933 19980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a3a4c931724d70a2f1591504560dd1" candidate_uuid: "c961b52c282947568f786b5e5f578bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df8e9777614e58aa903a7f2504812f"
I20260929 01:00:51.333020 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a3a4c931724d70a2f1591504560dd1" candidate_uuid: "c961b52c282947568f786b5e5f578bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0be499665a64ac38cd4f9b43aa254b7"
I20260929 01:00:51.333062 19980 raft_consensus.cc:3037] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:51.333139 19846 raft_consensus.cc:3037] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:51.334393 19846 raft_consensus.cc:2454] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c961b52c282947568f786b5e5f578bc5 in term 1.
I20260929 01:00:51.334393 19980 raft_consensus.cc:2454] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c961b52c282947568f786b5e5f578bc5 in term 1.
I20260929 01:00:51.334717 19645 leader_election.cc:302] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df8e9777614e58aa903a7f2504812f, c961b52c282947568f786b5e5f578bc5; no voters:
I20260929 01:00:51.334954 20051 raft_consensus.cc:2781] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:51.335194 20051 raft_consensus.cc:686] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [term 1 LEADER]: Becoming Leader. State: Replica: c961b52c282947568f786b5e5f578bc5, State: Running, Role: LEADER
I20260929 01:00:51.335546 20051 consensus_queue.cc:237] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260929 01:00:51.338610 19567 catalog_manager.cc:5495] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c961b52c282947568f786b5e5f578bc5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c961b52c282947568f786b5e5f578bc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c961b52c282947568f786b5e5f578bc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } health_report { overall_health: HEALTHY } } }
I20260929 01:00:51.404479 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.400885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.0.98.1:33605)
0929 01:00:51.401033 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.401038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.401054 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:00:51.401158 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.401160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.401215 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.401339 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:51.401596 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.402097 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.402998 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.403214 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.403329 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.403649 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:51.404069 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:51.404210 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.404253 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.404310 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:00:51.410540 19980 raft_consensus.cc:1261] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 1 FOLLOWER]: Refusing update from remote peer c961b52c282947568f786b5e5f578bc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:51.410557 19846 raft_consensus.cc:1261] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 1 FOLLOWER]: Refusing update from remote peer c961b52c282947568f786b5e5f578bc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:51.411237 20054 consensus_queue.cc:1035] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df8e9777614e58aa903a7f2504812f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:51.411437 20051 consensus_queue.cc:1035] T 35a3a4c931724d70a2f1591504560dd1 P c961b52c282947568f786b5e5f578bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0be499665a64ac38cd4f9b43aa254b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:51.419065 20057 mvcc.cc:205] Tried to move back new op lower bound from 7334476396170125312 to 7334476395873140736. Current Snapshot: MvccSnapshot[applied={T|T < 7334476396170125312}]
I20260929 01:00:51.420212 20058 mvcc.cc:205] Tried to move back new op lower bound from 7334476396170125312 to 7334476395873140736. Current Snapshot: MvccSnapshot[applied={T|T < 7334476396170125312}]
I20260929 01:00:51.428267 19566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:51.428407 19566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:51.430801 19566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3a9199910fc40c592516bd0317b1140: Sending DeleteTablet for 3 replicas of tablet 35a3a4c931724d70a2f1591504560dd1
I20260929 01:00:51.431537 19960 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a3a4c931724d70a2f1591504560dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47784
I20260929 01:00:51.431704 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a3a4c931724d70a2f1591504560dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43894
I20260929 01:00:51.431674 19826 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a3a4c931724d70a2f1591504560dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54394
I20260929 01:00:51.432082 20066 tablet_replica.cc:330] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: stopping tablet replica
I20260929 01:00:51.432391 20066 raft_consensus.cc:2229] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:51.432683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 19629
I20260929 01:00:51.432716 20066 raft_consensus.cc:2258] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:51.434435 20068 tablet_replica.cc:330] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: stopping tablet replica
I20260929 01:00:51.434742 20068 raft_consensus.cc:2229] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:51.435058 20068 raft_consensus.cc:2258] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:51.435738 20066 ts_tablet_manager.cc:1905] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:51.436466 20068 ts_tablet_manager.cc:1905] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:51.438066 20066 ts_tablet_manager.cc:1918] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:51.438180 20066 log.cc:1196] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/35a3a4c931724d70a2f1591504560dd1
I20260929 01:00:51.438505 20066 ts_tablet_manager.cc:1939] T 35a3a4c931724d70a2f1591504560dd1 P 97df8e9777614e58aa903a7f2504812f: Deleting consensus metadata
I20260929 01:00:51.439574 19551 catalog_manager.cc:4841] TS 97df8e9777614e58aa903a7f2504812f (127.0.98.3:38655): tablet 35a3a4c931724d70a2f1591504560dd1 (table test-table [id=d5ff0a025c7b42bcb8f9f7ecd5b70c28]) successfully deleted
I20260929 01:00:51.439747 20068 ts_tablet_manager.cc:1918] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:51.439846 20068 log.cc:1196] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643398448434-392-0/minicluster-data/ts-1/wal/wals/35a3a4c931724d70a2f1591504560dd1
I20260929 01:00:51.440167 20068 ts_tablet_manager.cc:1939] T 35a3a4c931724d70a2f1591504560dd1 P a0be499665a64ac38cd4f9b43aa254b7: Deleting consensus metadata
W20260929 01:00:51.440375 19554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33605 (error 108)
I20260929 01:00:51.440536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 19763
I20260929 01:00:51.441524 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.441206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33605 (local address 127.0.0.1:58720)
0929 01:00:51.441369 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.441443 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 01:00:51.441700 19554 catalog_manager.cc:4584] TS c961b52c282947568f786b5e5f578bc5 (127.0.98.1:33605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35a3a4c931724d70a2f1591504560dd1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33605: connect: Connection refused (error 111)
I20260929 01:00:51.447057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 19897
I20260929 01:00:51.447662 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.447335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37717 (local address 127.0.0.1:48848)
0929 01:00:51.447511 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.447568 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 01:00:51.452497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 19535
2026-09-29T01:00:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2495 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T01:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:51Z Disabled control of system clock
I20260929 01:00:51.483078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:00:51.590689 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:51.590940 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:51.590974 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:51.594890 20073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:51.594937 20073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:51.594956 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:51.594974 20073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:51.594990 20073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:51.599641 20073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:51.600661 20073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:51.601948 20073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:51.610220 20079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:51.610296 20078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:51.610311 20081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:51.610448 20073 server_base.cc:1034] running on GCE node
I20260929 01:00:51.611359 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:51.612444 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:51.613610 20073 hybrid_clock.cc:648] HybridClock initialized: now 1790643651613597 us; error 31 us; skew 500 ppm
I20260929 01:00:51.615506 20073 webserver.cc:460] Webserver started at http://127.0.98.62:44303/ using document root <none> and password file <none>
I20260929 01:00:51.616080 20073 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:51.616139 20073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:51.616354 20073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:51.618496 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "d0d9487a3b68424096004443e441586f"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-30s6"
server_key: "a27fcbf3ec73bbba610ef993649a1717"
server_key_iv: "a1a2339a1b3e880a44b783c291af9f70"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.619063 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0d9487a3b68424096004443e441586f"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-30s6"
server_key: "a27fcbf3ec73bbba610ef993649a1717"
server_key_iv: "a1a2339a1b3e880a44b783c291af9f70"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.622640 20073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:00:51.624964 20087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.626287 20073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:51.626417 20073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "d0d9487a3b68424096004443e441586f"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-30s6"
server_key: "a27fcbf3ec73bbba610ef993649a1717"
server_key_iv: "a1a2339a1b3e880a44b783c291af9f70"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.626525 20073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:51.641503 20073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:51.642197 20073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:51.642402 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:51.649281 20073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36633
I20260929 01:00:51.649286 20139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36633 every 8 connection(s)
I20260929 01:00:51.650216 20073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:51.653204 20140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:51.658444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 20073
I20260929 01:00:51.658604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:51.658730 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f: Bootstrap starting.
I20260929 01:00:51.658923 392 external_mini_cluster.cc:1428] Setting key 8855e1d9c65991904b24d3b94eb03d3d
I20260929 01:00:51.661303 20140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:51.662089 20140 log.cc:824] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:51.664059 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f: No bootstrap required, opened a new log
I20260929 01:00:51.667930 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.660279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.0.98.62:36633)
0929 01:00:51.660935 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.660949 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.660985 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:00:51.661663 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.661681 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.662081 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.662478 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:51.662894 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.663828 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.664769 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.665230 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.665409 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.666281 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:51.666286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:51.666298 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:51.666659 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:51.666710 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:51.666729 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:51.666849 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.666992 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.667236 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260929 01:00:51.667923 20140 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9487a3b68424096004443e441586f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260929 01:00:51.668231 20140 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:51.668334 20140 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d9487a3b68424096004443e441586f, State: Initialized, Role: FOLLOWER
I20260929 01:00:51.668745 20140 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9487a3b68424096004443e441586f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260929 01:00:51.668896 20140 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:51.668968 20140 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:51.669073 20140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:51.670013 20140 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9487a3b68424096004443e441586f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260929 01:00:51.670357 20140 leader_election.cc:302] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d9487a3b68424096004443e441586f; no voters:
I20260929 01:00:51.670663 20140 leader_election.cc:288] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:51.670796 20145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:51.671049 20145 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [term 1 LEADER]: Becoming Leader. State: Replica: d0d9487a3b68424096004443e441586f, State: Running, Role: LEADER
I20260929 01:00:51.671340 20145 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9487a3b68424096004443e441586f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260929 01:00:51.671805 20140 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:51.672971 20146 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d9487a3b68424096004443e441586f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9487a3b68424096004443e441586f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260929 01:00:51.673092 20146 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:51.673375 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d9487a3b68424096004443e441586f. Latest consensus state: current_term: 1 leader_uuid: "d0d9487a3b68424096004443e441586f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9487a3b68424096004443e441586f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260929 01:00:51.673461 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:51.674292 20154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:51.676811 20154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:51.682168 20154 catalog_manager.cc:1347] Generated new cluster ID: dd5da5e080a64a7598aff52460d48f72
I20260929 01:00:51.682267 20154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:51.691607 20154 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:51.692507 20154 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:51.699328 20154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f: Generated new TSK 0
I20260929 01:00:51.700018 20154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:51.706439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--builtin_ntp_servers=127.0.98.20:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:00:51.819597 20164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:51.819851 20164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:51.819919 20164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:51.823940 20164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:51.823997 20164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:51.824052 20164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:00:51.829023 20164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:51.830214 20164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:51.831671 20164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:51.841544 20170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:51.841612 20169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:51.841730 20172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:51.841619 20164 server_base.cc:1034] running on GCE node
I20260929 01:00:51.842115 20164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:51.842883 20164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:51.844064 20164 hybrid_clock.cc:648] HybridClock initialized: now 1790643651844030 us; error 46 us; skew 500 ppm
I20260929 01:00:51.849748 20164 webserver.cc:460] Webserver started at http://127.0.98.1:39577/ using document root <none> and password file <none>
I20260929 01:00:51.850443 20164 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:51.850512 20164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:51.850757 20164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:51.852869 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/data/instance:
uuid: "527d23b932d34d6d8553407e7693cf98"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-30s6"
server_key: "8b251598b8d47d92af97743019e33f48"
server_key_iv: "022cc60ef161a03f93015204651aac9e"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.853431 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/wal/instance:
uuid: "527d23b932d34d6d8553407e7693cf98"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-30s6"
server_key: "8b251598b8d47d92af97743019e33f48"
server_key_iv: "022cc60ef161a03f93015204651aac9e"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.857161 20164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:00:51.859576 20178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.860843 20164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:00:51.860983 20164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/wal
uuid: "527d23b932d34d6d8553407e7693cf98"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-30s6"
server_key: "8b251598b8d47d92af97743019e33f48"
server_key_iv: "022cc60ef161a03f93015204651aac9e"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.861099 20164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:51.882081 20164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:51.882834 20164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:51.883041 20164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:51.883669 20164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:51.884600 20164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:51.884672 20164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.884743 20164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:51.884791 20164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.893185 20164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38401
I20260929 01:00:51.893220 20291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38401 every 8 connection(s)
I20260929 01:00:51.894096 20164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:51.903005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 20164
I20260929 01:00:51.903162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:51.903474 392 external_mini_cluster.cc:1428] Setting key a10f3fb292fe57b885bd5e1a33c91562
I20260929 01:00:51.903484 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.895854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54043 (local address 127.0.98.62:36633)
0929 01:00:51.895981 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.895985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.897048 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 01:00:51.898416 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.898419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.898464 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.898539 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:51.899486 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.900374 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.901458 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.901672 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.901747 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.902889 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:51.902893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:51.902896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:51.902911 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:51.902937 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:51.902948 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:51.903035 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.903264 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.903332 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:00:51.904392 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.896192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.1:54043)
0929 01:00:51.896888 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.896930 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:51.898170 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:51.898672 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:51.898678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:51.898759 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:51.899311 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.899328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.900471 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.900473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.901357 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.901365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.901799 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.901802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.901946 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.902623 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:51.902624 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:51.902674 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:51.902737 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:51.903030 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:51.903035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:51.903038 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 01:00:51.903188 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 01:00:51.903448 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":103,"threads_started":1}
I20260929 01:00:51.905505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--builtin_ntp_servers=127.0.98.20:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:00:51.905560 20292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260929 01:00:51.905800 20292 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:51.906371 20292 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260929 01:00:51.907822 20104 ts_manager.cc:194] Registered new tserver with Master: 527d23b932d34d6d8553407e7693cf98 (127.0.98.1:38401)
I20260929 01:00:51.908952 20104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54043
W20260929 01:00:52.015645 20295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:52.015980 20295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:52.016058 20295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:52.020123 20295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:52.020211 20295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:52.020303 20295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:00:52.025107 20295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:52.026206 20295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:52.027632 20295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:52.037600 20301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:52.037600 20303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:52.037786 20295 server_base.cc:1034] running on GCE node
W20260929 01:00:52.038071 20300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:52.038501 20295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:52.039197 20295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:52.040388 20295 hybrid_clock.cc:648] HybridClock initialized: now 1790643652040375 us; error 34 us; skew 500 ppm
I20260929 01:00:52.042389 20295 webserver.cc:460] Webserver started at http://127.0.98.2:36595/ using document root <none> and password file <none>
I20260929 01:00:52.042977 20295 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:52.043071 20295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:52.043306 20295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:52.045360 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/data/instance:
uuid: "427925b4c0c04819994b1173a877a072"
format_stamp: "Formatted at 2026-09-29 01:00:52 on dist-test-slave-30s6"
server_key: "5e297499cca1c5ca607f001739590756"
server_key_iv: "a52d06f0b01907bd45862b47480e3c7d"
server_key_version: "encryptionkey@0"
I20260929 01:00:52.045933 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/wal/instance:
uuid: "427925b4c0c04819994b1173a877a072"
format_stamp: "Formatted at 2026-09-29 01:00:52 on dist-test-slave-30s6"
server_key: "5e297499cca1c5ca607f001739590756"
server_key_iv: "a52d06f0b01907bd45862b47480e3c7d"
server_key_version: "encryptionkey@0"
I20260929 01:00:52.049569 20295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:00:52.052156 20309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:52.053354 20295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:00:52.053491 20295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/wal
uuid: "427925b4c0c04819994b1173a877a072"
format_stamp: "Formatted at 2026-09-29 01:00:52 on dist-test-slave-30s6"
server_key: "5e297499cca1c5ca607f001739590756"
server_key_iv: "a52d06f0b01907bd45862b47480e3c7d"
server_key_version: "encryptionkey@0"
I20260929 01:00:52.053610 20295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:52.063031 20295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:52.063647 20295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:52.063865 20295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:52.064420 20295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:52.065332 20295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:52.065379 20295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:52.065416 20295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:52.065431 20295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:52.074829 20295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32955
I20260929 01:00:52.074852 20422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32955 every 8 connection(s)
I20260929 01:00:52.075814 20295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:52.081599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 20295
I20260929 01:00:52.081704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:52.082016 392 external_mini_cluster.cc:1428] Setting key 74035eb3e68befe04a552a3d13732d7c
I20260929 01:00:52.084030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--builtin_ntp_servers=127.0.98.20:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:00:52.085542 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.077582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54689 (local address 127.0.98.62:36633)
0929 01:00:52.077726 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.077730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.078813 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 01:00:52.080210 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.080213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.080269 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.080392 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:52.081260 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.081900 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.083463 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.083757 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.083903 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.085011 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:52.085018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:52.085024 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:52.085042 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:52.085069 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:52.085083 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:52.085170 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.085380 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.085439 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:00:52.086755 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.077986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.2:54689)
0929 01:00:52.078642 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.078710 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:52.079998 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:52.080557 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:52.080563 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:52.080652 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:52.081117 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.081129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.082032 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.082035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.083150 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.083162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.083934 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.083937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.084072 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.084759 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:52.084760 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:52.084799 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:52.084858 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:52.085201 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:52.085212 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:52.085219 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:00:52.085362 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 01:00:52.085725 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":138,"threads_started":1}
I20260929 01:00:52.087997 20423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260929 01:00:52.088274 20423 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:52.088841 20423 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260929 01:00:52.089952 20104 ts_manager.cc:194] Registered new tserver with Master: 427925b4c0c04819994b1173a877a072 (127.0.98.2:32955)
I20260929 01:00:52.090853 20104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54689
W20260929 01:00:52.195617 20426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:52.195859 20426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:52.195896 20426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:52.199872 20426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:52.199921 20426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:52.199973 20426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:00:52.204603 20426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:52.205562 20426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:52.206954 20426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:52.216315 20432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:52.216398 20431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:52.216323 20434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:52.216761 20426 server_base.cc:1034] running on GCE node
I20260929 01:00:52.217207 20426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:52.217823 20426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:52.219007 20426 hybrid_clock.cc:648] HybridClock initialized: now 1790643652218988 us; error 28 us; skew 500 ppm
I20260929 01:00:52.220821 20426 webserver.cc:460] Webserver started at http://127.0.98.3:44249/ using document root <none> and password file <none>
I20260929 01:00:52.221391 20426 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:52.221482 20426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:52.221707 20426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:52.223794 20426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9bd525b103942cc85c9e99d7eec2642"
format_stamp: "Formatted at 2026-09-29 01:00:52 on dist-test-slave-30s6"
server_key: "26cf3ee1347ed08bc0aaa951a113b7e4"
server_key_iv: "4490489310268d6a9e549174023d93a6"
server_key_version: "encryptionkey@0"
I20260929 01:00:52.224350 20426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9bd525b103942cc85c9e99d7eec2642"
format_stamp: "Formatted at 2026-09-29 01:00:52 on dist-test-slave-30s6"
server_key: "26cf3ee1347ed08bc0aaa951a113b7e4"
server_key_iv: "4490489310268d6a9e549174023d93a6"
server_key_version: "encryptionkey@0"
I20260929 01:00:52.228046 20426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:00:52.230577 20440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:52.231789 20426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:00:52.231921 20426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/wal
uuid: "f9bd525b103942cc85c9e99d7eec2642"
format_stamp: "Formatted at 2026-09-29 01:00:52 on dist-test-slave-30s6"
server_key: "26cf3ee1347ed08bc0aaa951a113b7e4"
server_key_iv: "4490489310268d6a9e549174023d93a6"
server_key_version: "encryptionkey@0"
I20260929 01:00:52.232031 20426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:52.244441 20426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:52.245090 20426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:52.245280 20426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:52.245839 20426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:52.246809 20426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:52.246879 20426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:52.246950 20426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:52.246994 20426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:52.255780 20426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34595
I20260929 01:00:52.255800 20553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34595 every 8 connection(s)
I20260929 01:00:52.256675 20426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:52.260807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 20426
I20260929 01:00:52.260951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:52.261212 392 external_mini_cluster.cc:1428] Setting key 0ce514cb1e54faa1ea80837b8b399dce
I20260929 01:00:52.266229 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.258689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51065 (local address 127.0.98.62:36633)
0929 01:00:52.258831 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.258834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.259804 (+ 970us) server_negotiation.cc:407] Connection header received
0929 01:00:52.261315 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.261319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.261383 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.261496 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:52.262458 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.262991 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.264164 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.264390 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.264526 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.265710 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:52.265714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:52.265716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:52.265735 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:52.265753 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:52.265761 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:52.265839 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.266003 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.266061 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:00:52.267093 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.259005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.3:51065)
0929 01:00:52.259654 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.259693 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:52.261101 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:52.261677 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:52.261685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:52.261771 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:52.262319 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.262335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.263121 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.263126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.264031 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.264041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.264577 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.264582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.264756 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.265393 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:52.265395 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:52.265435 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:52.265504 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:52.265855 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:52.265864 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:52.265870 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:00:52.266002 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 01:00:52.266313 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260929 01:00:52.268131 20554 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260929 01:00:52.268338 20554 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:52.268885 20554 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260929 01:00:52.269899 20104 ts_manager.cc:194] Registered new tserver with Master: f9bd525b103942cc85c9e99d7eec2642 (127.0.98.3:34595)
I20260929 01:00:52.270509 20104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51065
I20260929 01:00:52.274858 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:52.280858 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.276819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.0.98.62:36633)
0929 01:00:52.276999 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.277004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.277169 (+ 165us) server_negotiation.cc:407] Connection header received
0929 01:00:52.277320 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.277323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.277368 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.277453 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:52.277746 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.278313 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.279244 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.279463 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.279566 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.280426 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:52.280430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:52.280432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:52.280443 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:52.280461 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:52.280469 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:52.280550 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.280661 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.280713 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:00:52.283488 20104 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:52.286478 20104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:52.300673 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.295374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.0.98.1:38401)
0929 01:00:52.295859 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.295873 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.296019 (+ 146us) server_negotiation.cc:407] Connection header received
0929 01:00:52.296123 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.296129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.296330 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.296490 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:52.296754 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.297969 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.299499 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.300031 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.300154 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.300368 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.300422 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.300515 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":119,"threads_started":1}
I20260929 01:00:52.300838 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.294524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34595 (local address 127.0.0.1:45912)
0929 01:00:52.295355 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.295406 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:52.295547 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:52.296807 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:52.296809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:52.296827 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:52.296924 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.296934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.297935 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.297939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.299381 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.299389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.300362 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.300366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.300461 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.300489 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:00:52.300541 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 01:00:52.300621 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":76,"spinlock_wait_cycles":9344,"thread_start_us":114,"threads_started":1}
I20260929 01:00:52.300877 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.294607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32955 (local address 127.0.0.1:59548)
0929 01:00:52.295356 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.295405 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:52.295547 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:52.296642 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:52.296645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:52.296661 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:52.296767 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.296774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.298148 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.298156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.299229 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.299239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.300528 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.300534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.300611 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.300629 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 01:00:52.300693 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 01:00:52.300751 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":1,"spinlock_wait_cycles":6272,"thread_start_us":107,"threads_started":1}
I20260929 01:00:52.300669 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.294880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38401 (local address 127.0.0.1:40212)
0929 01:00:52.295875 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.295912 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:52.296013 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:52.296478 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:52.296484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:52.296507 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:52.296630 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.296636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.298085 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.298088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.299353 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.299362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.300163 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.300166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.300263 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.300299 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:00:52.300398 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:00:52.300511 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":36,"thread_start_us":99,"threads_started":1}
I20260929 01:00:52.301208 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.294658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.0.98.3:34595)
0929 01:00:52.296290 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.296305 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.296328 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:00:52.296456 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.296460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.296643 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.296823 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:52.297052 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.297813 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.299500 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.300238 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.300735 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.300898 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.300950 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.301051 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1504,"thread_start_us":149,"threads_started":1}
I20260929 01:00:52.301311 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.295330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.0.98.2:32955)
0929 01:00:52.296120 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.296132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.296151 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:00:52.296213 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.296217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.296354 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.296489 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:52.296903 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.298018 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.299365 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.300158 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.300275 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.300471 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.301049 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.301150 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":104,"threads_started":1}
I20260929 01:00:52.303910 20226 tablet_service.cc:1467] Processing CreateTablet for tablet 17ccf72e70a04a12adb0f27d30f4b23a (DEFAULT_TABLE table=test-table [id=3e75b1ff91544854bf40f171f78ae0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:52.304039 20357 tablet_service.cc:1467] Processing CreateTablet for tablet 17ccf72e70a04a12adb0f27d30f4b23a (DEFAULT_TABLE table=test-table [id=3e75b1ff91544854bf40f171f78ae0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:52.304040 20488 tablet_service.cc:1467] Processing CreateTablet for tablet 17ccf72e70a04a12adb0f27d30f4b23a (DEFAULT_TABLE table=test-table [id=3e75b1ff91544854bf40f171f78ae0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:52.304934 20226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ccf72e70a04a12adb0f27d30f4b23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:52.305024 20488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ccf72e70a04a12adb0f27d30f4b23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:52.305024 20357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ccf72e70a04a12adb0f27d30f4b23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:52.310886 20571 tablet_bootstrap.cc:492] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98: Bootstrap starting.
I20260929 01:00:52.312611 20573 tablet_bootstrap.cc:492] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Bootstrap starting.
I20260929 01:00:52.312990 20572 tablet_bootstrap.cc:492] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072: Bootstrap starting.
I20260929 01:00:52.313283 20571 tablet_bootstrap.cc:654] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:52.314224 20571 log.cc:824] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:52.314805 20573 tablet_bootstrap.cc:654] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:52.315049 20572 tablet_bootstrap.cc:654] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:52.315574 20573 log.cc:824] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:52.315826 20572 log.cc:824] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:52.316452 20571 tablet_bootstrap.cc:492] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98: No bootstrap required, opened a new log
I20260929 01:00:52.316644 20571 ts_tablet_manager.cc:1397] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:52.317503 20572 tablet_bootstrap.cc:492] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072: No bootstrap required, opened a new log
I20260929 01:00:52.317566 20573 tablet_bootstrap.cc:492] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: No bootstrap required, opened a new log
I20260929 01:00:52.317711 20572 ts_tablet_manager.cc:1397] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 01:00:52.317742 20573 ts_tablet_manager.cc:1397] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:00:52.321056 20572 raft_consensus.cc:348] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.321071 20571 raft_consensus.cc:348] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.321312 20572 raft_consensus.cc:374] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:52.321323 20571 raft_consensus.cc:374] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:52.321365 20572 raft_consensus.cc:729] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427925b4c0c04819994b1173a877a072, State: Initialized, Role: FOLLOWER
I20260929 01:00:52.321380 20571 raft_consensus.cc:729] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527d23b932d34d6d8553407e7693cf98, State: Initialized, Role: FOLLOWER
I20260929 01:00:52.321815 20572 consensus_queue.cc:260] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.321830 20571 consensus_queue.cc:260] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.321807 20573 raft_consensus.cc:348] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.322029 20573 raft_consensus.cc:374] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:52.322096 20573 raft_consensus.cc:729] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bd525b103942cc85c9e99d7eec2642, State: Initialized, Role: FOLLOWER
I20260929 01:00:52.322533 20573 consensus_queue.cc:260] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.322841 20572 ts_tablet_manager.cc:1428] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260929 01:00:52.322867 20292 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
I20260929 01:00:52.323076 20423 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
I20260929 01:00:52.323359 20573 ts_tablet_manager.cc:1428] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:00:52.323374 20554 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
I20260929 01:00:52.322860 20571 ts_tablet_manager.cc:1428] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 01:00:52.326719 20424 tablet.cc:2369] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:52.395248 20293 tablet.cc:2369] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:52.484040 20578 raft_consensus.cc:482] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:52.484280 20578 raft_consensus.cc:504] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.485549 20578 leader_election.cc:288] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9bd525b103942cc85c9e99d7eec2642 (127.0.98.3:34595), 427925b4c0c04819994b1173a877a072 (127.0.98.2:32955)
I20260929 01:00:52.491119 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.485946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59473 (local address 127.0.98.2:32955)
0929 01:00:52.486086 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.486093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.486399 (+ 306us) server_negotiation.cc:407] Connection header received
0929 01:00:52.486477 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.486481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.486541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.486668 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:52.487100 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.487915 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.489828 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.490446 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.490557 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.490617 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.490919 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.490990 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:00:52.491119 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.485888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37053 (local address 127.0.98.3:34595)
0929 01:00:52.486026 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.486032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.486343 (+ 311us) server_negotiation.cc:407] Connection header received
0929 01:00:52.486489 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.486492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.486541 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.486642 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:52.487100 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.487903 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.489822 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.490454 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.490557 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.490617 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.490919 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.490990 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:00:52.491179 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.485800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34595 (local address 127.0.98.1:37053)
0929 01:00:52.486220 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.486236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:52.486380 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:52.486666 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:52.486669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:52.486699 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:52.486907 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.486916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.488029 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.488033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.489684 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.489698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.490577 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.490581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.490752 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.490788 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:00:52.490873 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 01:00:52.490980 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":326,"spinlock_wait_cycles":7296,"thread_start_us":73,"threads_started":1}
I20260929 01:00:52.491753 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.485828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32955 (local address 127.0.98.1:59473)
0929 01:00:52.486304 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.486318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:52.486388 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:52.486666 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:52.486668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:52.486694 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:52.486917 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.486923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.488030 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.488033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.489691 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:52.489698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.490577 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.490581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:52.490742 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.490813 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 01:00:52.490883 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 01:00:52.490983 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":315,"spinlock_wait_cycles":2816,"thread_start_us":96,"threads_started":1}
I20260929 01:00:52.492074 20508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ccf72e70a04a12adb0f27d30f4b23a" candidate_uuid: "527d23b932d34d6d8553407e7693cf98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9bd525b103942cc85c9e99d7eec2642" is_pre_election: true
I20260929 01:00:52.492431 20508 raft_consensus.cc:2454] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527d23b932d34d6d8553407e7693cf98 in term 0.
I20260929 01:00:52.492432 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ccf72e70a04a12adb0f27d30f4b23a" candidate_uuid: "527d23b932d34d6d8553407e7693cf98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427925b4c0c04819994b1173a877a072" is_pre_election: true
I20260929 01:00:52.492734 20377 raft_consensus.cc:2454] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527d23b932d34d6d8553407e7693cf98 in term 0.
I20260929 01:00:52.492949 20179 leader_election.cc:302] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 527d23b932d34d6d8553407e7693cf98, f9bd525b103942cc85c9e99d7eec2642; no voters:
I20260929 01:00:52.493255 20578 raft_consensus.cc:2781] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:52.493355 20578 raft_consensus.cc:482] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:52.493417 20578 raft_consensus.cc:3037] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:52.494469 20578 raft_consensus.cc:504] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.494832 20578 leader_election.cc:288] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [CANDIDATE]: Term 1 election: Requested vote from peers f9bd525b103942cc85c9e99d7eec2642 (127.0.98.3:34595), 427925b4c0c04819994b1173a877a072 (127.0.98.2:32955)
I20260929 01:00:52.495193 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ccf72e70a04a12adb0f27d30f4b23a" candidate_uuid: "527d23b932d34d6d8553407e7693cf98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427925b4c0c04819994b1173a877a072"
I20260929 01:00:52.495332 20377 raft_consensus.cc:3037] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:52.495330 20508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ccf72e70a04a12adb0f27d30f4b23a" candidate_uuid: "527d23b932d34d6d8553407e7693cf98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9bd525b103942cc85c9e99d7eec2642"
I20260929 01:00:52.495476 20508 raft_consensus.cc:3037] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:52.496379 20377 raft_consensus.cc:2454] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d23b932d34d6d8553407e7693cf98 in term 1.
I20260929 01:00:52.496490 20508 raft_consensus.cc:2454] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d23b932d34d6d8553407e7693cf98 in term 1.
I20260929 01:00:52.496742 20181 leader_election.cc:302] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427925b4c0c04819994b1173a877a072, 527d23b932d34d6d8553407e7693cf98; no voters:
I20260929 01:00:52.496920 20578 raft_consensus.cc:2781] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:52.497157 20578 raft_consensus.cc:686] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [term 1 LEADER]: Becoming Leader. State: Replica: 527d23b932d34d6d8553407e7693cf98, State: Running, Role: LEADER
I20260929 01:00:52.497479 20578 consensus_queue.cc:237] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } }
I20260929 01:00:52.500603 20104 catalog_manager.cc:5495] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 527d23b932d34d6d8553407e7693cf98 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "527d23b932d34d6d8553407e7693cf98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d23b932d34d6d8553407e7693cf98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38401 } health_report { overall_health: HEALTHY } } }
W20260929 01:00:52.507652 20555 tablet.cc:2369] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:52.543454 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.540186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.0.98.1:38401)
0929 01:00:52.540321 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.540325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.540339 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:00:52.540422 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.540424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.540481 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.540573 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:52.540794 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.541279 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.542069 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.542336 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.542418 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.542663 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:52.543081 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:52.543171 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.543235 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.543282 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:00:52.549415 20508 raft_consensus.cc:1261] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 1 FOLLOWER]: Refusing update from remote peer 527d23b932d34d6d8553407e7693cf98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:52.549415 20377 raft_consensus.cc:1261] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 1 FOLLOWER]: Refusing update from remote peer 527d23b932d34d6d8553407e7693cf98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:52.550079 20582 consensus_queue.cc:1035] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427925b4c0c04819994b1173a877a072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:52.550309 20578 consensus_queue.cc:1035] T 17ccf72e70a04a12adb0f27d30f4b23a P 527d23b932d34d6d8553407e7693cf98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9bd525b103942cc85c9e99d7eec2642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:52.558101 20585 mvcc.cc:205] Tried to move back new op lower bound from 7334476400835305472 to 7334476400632733696. Current Snapshot: MvccSnapshot[applied={T|T < 7334476400835305472}]
I20260929 01:00:52.558275 20587 mvcc.cc:205] Tried to move back new op lower bound from 7334476400835305472 to 7334476400632733696. Current Snapshot: MvccSnapshot[applied={T|T < 7334476400835305472}]
I20260929 01:00:52.567268 20104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:52.567413 20104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:52.569619 20104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0d9487a3b68424096004443e441586f: Sending DeleteTablet for 3 replicas of tablet 17ccf72e70a04a12adb0f27d30f4b23a
I20260929 01:00:52.570431 20488 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ccf72e70a04a12adb0f27d30f4b23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:52 UTC) from {username='slave'} at 127.0.0.1:45912
I20260929 01:00:52.570452 20357 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ccf72e70a04a12adb0f27d30f4b23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:52 UTC) from {username='slave'} at 127.0.0.1:59548
I20260929 01:00:52.570642 20226 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ccf72e70a04a12adb0f27d30f4b23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:52 UTC) from {username='slave'} at 127.0.0.1:40212
I20260929 01:00:52.570945 20596 tablet_replica.cc:330] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: stopping tablet replica
I20260929 01:00:52.571173 20595 tablet_replica.cc:330] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072: stopping tablet replica
I20260929 01:00:52.571264 20596 raft_consensus.cc:2229] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:52.571456 20595 raft_consensus.cc:2229] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:52.571578 20596 raft_consensus.cc:2258] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:52.571691 20595 raft_consensus.cc:2258] T 17ccf72e70a04a12adb0f27d30f4b23a P 427925b4c0c04819994b1173a877a072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:52.572084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 20164
I20260929 01:00:52.573576 20596 ts_tablet_manager.cc:1905] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:52.576015 20596 ts_tablet_manager.cc:1918] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:52.576126 20596 log.cc:1196] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Deleting WAL directory at /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643398448434-392-0/minicluster-data/ts-2/wal/wals/17ccf72e70a04a12adb0f27d30f4b23a
I20260929 01:00:52.576423 20596 ts_tablet_manager.cc:1939] T 17ccf72e70a04a12adb0f27d30f4b23a P f9bd525b103942cc85c9e99d7eec2642: Deleting consensus metadata
I20260929 01:00:52.577512 20088 catalog_manager.cc:4841] TS f9bd525b103942cc85c9e99d7eec2642 (127.0.98.3:34595): tablet 17ccf72e70a04a12adb0f27d30f4b23a (table test-table [id=3e75b1ff91544854bf40f171f78ae0d4]) successfully deleted
I20260929 01:00:52.579151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 20295
W20260929 01:00:52.579128 20089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38401 (error 108)
I20260929 01:00:52.580631 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.580322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38401 (local address 127.0.0.1:40236)
0929 01:00:52.580473 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.580548 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 01:00:52.580807 20089 catalog_manager.cc:4584] TS 527d23b932d34d6d8553407e7693cf98 (127.0.98.1:38401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17ccf72e70a04a12adb0f27d30f4b23a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38401: connect: Connection refused (error 111)
I20260929 01:00:52.584345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 20426
I20260929 01:00:52.585832 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.584413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32955 (local address 127.0.0.1:59554)
0929 01:00:52.585697 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:52.585752 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1197}
I20260929 01:00:52.589700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 20073
2026-09-29T01:00:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1136 ms)
[----------] 6 tests from EncryptionPolicyTest (10909 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:52 dist-test-slave-30s6 krb5kdc[20601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:52 dist-test-slave-30s6 krb5kdc[20601](info): set up 2 sockets
Sep 29 01:00:52 dist-test-slave-30s6 krb5kdc[20601](info): commencing operation
krb5kdc: starting...
W20260929 01:00:53.634857 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:00:53 dist-test-slave-30s6 krb5kdc[20601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:00:53.757725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:53.869395 20614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:53.869688 20614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:53.869761 20614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:53.873898 20614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:53.873987 20614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:53.874028 20614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:53.874064 20614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:53.874099 20614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:53.879654 20614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:53.880751 20614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:53.881785 20614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:53.890257 20620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:53.890331 20619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:53.890257 20622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:53.890523 20614 server_base.cc:1034] running on GCE node
I20260929 01:00:53.891211 20614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:53.892325 20614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:53.893520 20614 hybrid_clock.cc:648] HybridClock initialized: now 1790643653893500 us; error 40 us; skew 500 ppm
Sep 29 01:00:53 dist-test-slave-30s6 krb5kdc[20601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:53.896327 20614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:00:53.897184 20614 webserver.cc:460] Webserver started at http://127.0.0.1:43563/ using document root <none> and password file <none>
I20260929 01:00:53.897735 20614 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:53.897826 20614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:53.898067 20614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:53.900112 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "8af81d90302a415fb2758a4dafb8a31b"
format_stamp: "Formatted at 2026-09-29 01:00:53 on dist-test-slave-30s6"
server_key: "086bb3bfc200b26ffdde78e4fafcee84"
server_key_iv: "463c71450ebcd9eb89c258e2092bbdaa"
server_key_version: "encryptionkey@0"
I20260929 01:00:53.900678 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "8af81d90302a415fb2758a4dafb8a31b"
format_stamp: "Formatted at 2026-09-29 01:00:53 on dist-test-slave-30s6"
server_key: "086bb3bfc200b26ffdde78e4fafcee84"
server_key_iv: "463c71450ebcd9eb89c258e2092bbdaa"
server_key_version: "encryptionkey@0"
I20260929 01:00:53.904384 20614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:00:53.906708 20629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:53.907833 20614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:53.907984 20614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "8af81d90302a415fb2758a4dafb8a31b"
format_stamp: "Formatted at 2026-09-29 01:00:53 on dist-test-slave-30s6"
server_key: "086bb3bfc200b26ffdde78e4fafcee84"
server_key_iv: "463c71450ebcd9eb89c258e2092bbdaa"
server_key_version: "encryptionkey@0"
I20260929 01:00:53.908094 20614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:53.928984 20614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:53.932461 20614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:53.932677 20614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:53.939440 20614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42725
I20260929 01:00:53.939471 20681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42725 every 8 connection(s)
I20260929 01:00:53.940351 20614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:53.943024 20682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:53.943661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 20614
I20260929 01:00:53.943784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:53.944067 392 external_mini_cluster.cc:1428] Setting key 22419995e82a9845d7f452ced0d6c4ae
I20260929 01:00:53.949700 20682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b: Bootstrap starting.
I20260929 01:00:53.952184 20682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:53.952911 20682 log.cc:824] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:00:53 dist-test-slave-30s6 krb5kdc[20601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:00:53.955134 20682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b: No bootstrap required, opened a new log
I20260929 01:00:53.959136 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:53.945448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55135 (local address 127.0.0.1:42725)
0929 01:00:53.945885 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:53.945904 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:53.945949 (+ 45us) server_negotiation.cc:407] Connection header received
0929 01:00:53.946774 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:53.946796 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:53.947219 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:53.947614 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:53.948276 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:53.949728 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:53.950650 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:53.951327 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:53.951490 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:53.953964 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:53.953970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:53.953985 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:53.954013 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:53.955920 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:53.956386 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:53.956391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:53.956397 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:53.956534 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:53.956826 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:53.956831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:53.956833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:53.957412 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:53.957616 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:53.957906 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 01:00:53.958145 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":106,"threads_started":1}
I20260929 01:00:53.959661 20682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af81d90302a415fb2758a4dafb8a31b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42725 } }
I20260929 01:00:53.959914 20682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:53.959976 20682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af81d90302a415fb2758a4dafb8a31b, State: Initialized, Role: FOLLOWER
I20260929 01:00:53.960466 20682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af81d90302a415fb2758a4dafb8a31b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42725 } }
I20260929 01:00:53.960620 20682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:53.960681 20682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:53.960772 20682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:53.961833 20682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af81d90302a415fb2758a4dafb8a31b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42725 } }
I20260929 01:00:53.962217 20682 leader_election.cc:302] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8af81d90302a415fb2758a4dafb8a31b; no voters:
I20260929 01:00:53.962275 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:00:53.962543 20682 leader_election.cc:288] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:53.962673 20687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:53.962916 20687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [term 1 LEADER]: Becoming Leader. State: Replica: 8af81d90302a415fb2758a4dafb8a31b, State: Running, Role: LEADER
I20260929 01:00:53.963292 20687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af81d90302a415fb2758a4dafb8a31b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42725 } }
I20260929 01:00:53.963864 20682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:53.964897 20688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8af81d90302a415fb2758a4dafb8a31b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af81d90302a415fb2758a4dafb8a31b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42725 } } }
I20260929 01:00:53.965039 20688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:53.965155 20689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8af81d90302a415fb2758a4dafb8a31b. Latest consensus state: current_term: 1 leader_uuid: "8af81d90302a415fb2758a4dafb8a31b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af81d90302a415fb2758a4dafb8a31b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42725 } } }
I20260929 01:00:53.965236 20689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:53.965615 20701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:53.969426 20701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:53.974224 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:53.964452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58837 (local address 127.0.0.1:42725)
0929 01:00:53.964565 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:53.964572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:53.964841 (+ 269us) server_negotiation.cc:407] Connection header received
0929 01:00:53.965027 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:53.965030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:53.965083 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:53.965157 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:53.966133 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:53.966882 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:53.967973 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:53.968285 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:53.968392 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:53.970340 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:53.970344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:53.970347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:53.970382 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:53.972047 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:53.972717 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:53.972721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:53.972723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:53.972825 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:53.973201 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:53.973204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:53.973206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:53.973437 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:53.973577 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:53.973951 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 01:00:53.974058 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 01:00:53.975135 20701 catalog_manager.cc:1347] Generated new cluster ID: 62cf6be9bf3046bdaae951f2bc038b35
I20260929 01:00:53.975198 20701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:53.987426 20701 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:53.988292 20701 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:53.993903 20701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8af81d90302a415fb2758a4dafb8a31b: Generated new TSK 0
I20260929 01:00:53.994599 20701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:54.001780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 20614
W20260929 01:00:54.006978 20599 connection.cc:535] client connection to 127.0.0.1:42725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42725): Connection reset by peer (error 104)
2026-09-29T01:00:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:54 dist-test-slave-30s6 krb5kdc[20713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:54 dist-test-slave-30s6 krb5kdc[20713](info): set up 2 sockets
Sep 29 01:00:54 dist-test-slave-30s6 krb5kdc[20713](info): commencing operation
krb5kdc: starting...
W20260929 01:00:55.034303 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:00:55 dist-test-slave-30s6 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:00:55.157182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:55.266619 20726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:55.266923 20726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:55.267000 20726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:55.270936 20726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:55.271021 20726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:55.271064 20726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:55.271121 20726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:55.271159 20726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:55.276503 20726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:55.277551 20726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:55.278599 20726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:55.286945 20734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:55.287014 20732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:55.286983 20731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:55.287025 20726 server_base.cc:1034] running on GCE node
I20260929 01:00:55.287740 20726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:55.288882 20726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:55.290073 20726 hybrid_clock.cc:648] HybridClock initialized: now 1790643655290043 us; error 51 us; skew 500 ppm
Sep 29 01:00:55 dist-test-slave-30s6 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:55.292948 20726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:00:55.293795 20726 webserver.cc:460] Webserver started at http://127.0.0.1:38661/ using document root <none> and password file <none>
I20260929 01:00:55.294397 20726 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:55.294461 20726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:55.294629 20726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:55.296602 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "7cdb8b6d8761401f9ee3ab36bf63071d"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-30s6"
server_key: "2ff812712a5937c53b83ea1ccb563886"
server_key_iv: "f17071d6fbff8ab1b4d20c7b77bddcbb"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.297108 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "7cdb8b6d8761401f9ee3ab36bf63071d"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-30s6"
server_key: "2ff812712a5937c53b83ea1ccb563886"
server_key_iv: "f17071d6fbff8ab1b4d20c7b77bddcbb"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.300737 20726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:00:55.303309 20741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:55.304567 20726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:00:55.304697 20726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "7cdb8b6d8761401f9ee3ab36bf63071d"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-30s6"
server_key: "2ff812712a5937c53b83ea1ccb563886"
server_key_iv: "f17071d6fbff8ab1b4d20c7b77bddcbb"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.304804 20726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:55.335753 20726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:55.340408 20726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:55.340642 20726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:55.349602 20726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34645
I20260929 01:00:55.349619 20793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34645 every 8 connection(s)
I20260929 01:00:55.350870 20726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:55.353417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 20726
I20260929 01:00:55.353586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:55.353879 392 external_mini_cluster.cc:1428] Setting key 05d2385b00731def11a9c036e17c12ac
I20260929 01:00:55.355037 20794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:55.363873 20794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d: Bootstrap starting.
I20260929 01:00:55.365061 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:55.355784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46843 (local address 127.0.0.1:34645)
0929 01:00:55.356271 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:55.356306 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:55.356349 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:00:55.357271 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:55.357296 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:55.357739 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:55.358139 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:55.358624 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.360161 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.361032 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.361776 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.361924 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:55.362893 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:55.362898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:55.362941 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:55.363478 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:55.363547 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:55.363577 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:55.363727 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:55.363910 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 01:00:55.364066 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":163,"threads_started":1}
I20260929 01:00:55.367058 20794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:55.368001 20794 log.cc:824] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:55.368811 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:00:55.370345 20794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d: No bootstrap required, opened a new log
I20260929 01:00:55.374915 20794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb8b6d8761401f9ee3ab36bf63071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } }
I20260929 01:00:55.375523 20794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:55.375608 20794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdb8b6d8761401f9ee3ab36bf63071d, State: Initialized, Role: FOLLOWER
I20260929 01:00:55.376117 20794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb8b6d8761401f9ee3ab36bf63071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } }
I20260929 01:00:55.376323 20794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:55.376407 20794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:55.376555 20794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:55.376626 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:55.371391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52945 (local address 127.0.0.1:34645)
0929 01:00:55.371510 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:55.371514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:55.371841 (+ 327us) server_negotiation.cc:407] Connection header received
0929 01:00:55.372033 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:55.372037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:55.372094 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:55.372189 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:55.372513 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.373762 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.374670 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.375044 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.375164 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:55.376078 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:55.376082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:55.376085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:55.376099 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:55.376131 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:55.376150 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:55.376258 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:55.376422 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 01:00:55.376461 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:00:55.377777 20794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb8b6d8761401f9ee3ab36bf63071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } }
I20260929 01:00:55.378244 20794 leader_election.cc:302] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cdb8b6d8761401f9ee3ab36bf63071d; no voters:
I20260929 01:00:55.378654 20794 leader_election.cc:288] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:55.378756 20804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:55.379029 20804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [term 1 LEADER]: Becoming Leader. State: Replica: 7cdb8b6d8761401f9ee3ab36bf63071d, State: Running, Role: LEADER
I20260929 01:00:55.379377 20804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb8b6d8761401f9ee3ab36bf63071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } }
I20260929 01:00:55.380067 20794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:55.381102 20805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cdb8b6d8761401f9ee3ab36bf63071d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb8b6d8761401f9ee3ab36bf63071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } } }
I20260929 01:00:55.381266 20805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:55.381124 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cdb8b6d8761401f9ee3ab36bf63071d. Latest consensus state: current_term: 1 leader_uuid: "7cdb8b6d8761401f9ee3ab36bf63071d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb8b6d8761401f9ee3ab36bf63071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } } }
I20260929 01:00:55.381389 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:55.381697 20813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:55.384788 20813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:55.390522 20813 catalog_manager.cc:1347] Generated new cluster ID: 659d3013ce4e4b369c192dd652d9da37
I20260929 01:00:55.390587 20813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:55.408677 20813 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:55.410022 20813 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:55.417932 20813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cdb8b6d8761401f9ee3ab36bf63071d: Generated new TSK 0
I20260929 01:00:55.418820 20813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:55.429757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 20726
W20260929 01:00:55.435153 20711 connection.cc:535] client connection to 127.0.0.1:34645 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34645): Connection reset by peer (error 104)
2026-09-29T01:00:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:55 dist-test-slave-30s6 krb5kdc[20825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:55 dist-test-slave-30s6 krb5kdc[20825](info): set up 2 sockets
Sep 29 01:00:55 dist-test-slave-30s6 krb5kdc[20825](info): commencing operation
krb5kdc: starting...
W20260929 01:00:56.462302 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:00:56 dist-test-slave-30s6 krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:00:56.585965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:56.693509 20838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:56.693770 20838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:56.693813 20838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:56.697705 20838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:56.697759 20838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:56.697778 20838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:56.697795 20838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:56.697810 20838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:56.702996 20838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:56.703975 20838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:56.704931 20838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:56.713291 20844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:56.713302 20843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:56.713310 20838 server_base.cc:1034] running on GCE node
W20260929 01:00:56.713325 20846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:56.714068 20838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:56.715147 20838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:56.716337 20838 hybrid_clock.cc:648] HybridClock initialized: now 1790643656716319 us; error 30 us; skew 500 ppm
Sep 29 01:00:56 dist-test-slave-30s6 krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:56.719058 20838 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:00:56.719895 20838 webserver.cc:460] Webserver started at http://127.0.0.1:41885/ using document root <none> and password file <none>
I20260929 01:00:56.720448 20838 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:56.720539 20838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:56.720762 20838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:56.722810 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "4a986a4915ef404da6b2dc93e0c08ac1"
format_stamp: "Formatted at 2026-09-29 01:00:56 on dist-test-slave-30s6"
server_key: "bfce0e1cc22d550a151065ae8bdc050f"
server_key_iv: "48d7d8d4672ced8a85f1b0f55e6b930c"
server_key_version: "encryptionkey@0"
I20260929 01:00:56.723368 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a986a4915ef404da6b2dc93e0c08ac1"
format_stamp: "Formatted at 2026-09-29 01:00:56 on dist-test-slave-30s6"
server_key: "bfce0e1cc22d550a151065ae8bdc050f"
server_key_iv: "48d7d8d4672ced8a85f1b0f55e6b930c"
server_key_version: "encryptionkey@0"
I20260929 01:00:56.726949 20838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:56.729238 20853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:56.730414 20838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:56.730556 20838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "4a986a4915ef404da6b2dc93e0c08ac1"
format_stamp: "Formatted at 2026-09-29 01:00:56 on dist-test-slave-30s6"
server_key: "bfce0e1cc22d550a151065ae8bdc050f"
server_key_iv: "48d7d8d4672ced8a85f1b0f55e6b930c"
server_key_version: "encryptionkey@0"
I20260929 01:00:56.730671 20838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:56.735196 20838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:56.738669 20838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:56.738873 20838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:56.745682 20838 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38359
I20260929 01:00:56.745740 20905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38359 every 8 connection(s)
I20260929 01:00:56.746637 20838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:56.749353 20906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:56.751538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 20838
I20260929 01:00:56.751796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:56.752085 392 external_mini_cluster.cc:1428] Setting key 95e42436e8077f203f3a4f84a1f62f25
I20260929 01:00:56.755594 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1: Bootstrap starting.
I20260929 01:00:56.757550 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.753440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41955 (local address 127.0.0.1:38359)
0929 01:00:56.753853 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.753871 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.753925 (+ 54us) server_negotiation.cc:407] Connection header received
0929 01:00:56.754753 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.754775 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.755141 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.755487 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:56.756230 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:56.756236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:56.756250 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:56.756261 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:56.756320 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:56.756345 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:56.756476 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.756537 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.756679 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260929 01:00:56.758286 20906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:56.759150 20906 log.cc:824] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:56.760548 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:00:56.761473 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1: No bootstrap required, opened a new log
I20260929 01:00:56.764354 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.762751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47917 (local address 127.0.0.1:38359)
0929 01:00:56.762880 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.762884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.763105 (+ 221us) server_negotiation.cc:407] Connection header received
0929 01:00:56.763208 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.763210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.763238 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.763300 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:56.763965 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:56.763968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:56.763969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:56.763979 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:56.764017 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:56.764028 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:56.764095 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.764196 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.764225 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:00:56.765918 20906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986a4915ef404da6b2dc93e0c08ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 01:00:56.766151 20906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:56.766230 20906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a986a4915ef404da6b2dc93e0c08ac1, State: Initialized, Role: FOLLOWER
I20260929 01:00:56.766680 20906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986a4915ef404da6b2dc93e0c08ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 01:00:56.766837 20906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:56.766909 20906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:56.767009 20906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:56.767877 20906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986a4915ef404da6b2dc93e0c08ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 01:00:56.768210 20906 leader_election.cc:302] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a986a4915ef404da6b2dc93e0c08ac1; no voters:
I20260929 01:00:56.768478 20906 leader_election.cc:288] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:56.768594 20916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:56.768806 20916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [term 1 LEADER]: Becoming Leader. State: Replica: 4a986a4915ef404da6b2dc93e0c08ac1, State: Running, Role: LEADER
I20260929 01:00:56.769115 20916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986a4915ef404da6b2dc93e0c08ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 01:00:56.769853 20906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:56.771032 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a986a4915ef404da6b2dc93e0c08ac1. Latest consensus state: current_term: 1 leader_uuid: "4a986a4915ef404da6b2dc93e0c08ac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986a4915ef404da6b2dc93e0c08ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } } }
I20260929 01:00:56.771040 20917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a986a4915ef404da6b2dc93e0c08ac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986a4915ef404da6b2dc93e0c08ac1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } } }
I20260929 01:00:56.771209 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:56.771226 20917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1 [sys.catalog]: This master's current role is: LEADER
W20260929 01:00:56.774624 20932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:00:56.774714 20932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:00:56.774817 20925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:56.775704 20925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:56.781435 20925 catalog_manager.cc:1347] Generated new cluster ID: ac689ff4834747bb98ab8287c576a5a0
I20260929 01:00:56.781498 20925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:56.795872 20925 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:56.796207 20925 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:56.801116 20925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a986a4915ef404da6b2dc93e0c08ac1: Generated new TSK 0
I20260929 01:00:56.801755 20925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:56.810894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 20838
2026-09-29T01:00:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:56 dist-test-slave-30s6 krb5kdc[20937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:56 dist-test-slave-30s6 krb5kdc[20937](info): set up 2 sockets
Sep 29 01:00:56 dist-test-slave-30s6 krb5kdc[20937](info): commencing operation
krb5kdc: starting...
W20260929 01:00:57.854468 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:00:57 dist-test-slave-30s6 krb5kdc[20937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:00:57.975229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:58.082894 20950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:58.083158 20950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:58.083199 20950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:58.087033 20950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:58.087085 20950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:58.087103 20950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:58.087119 20950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:58.087136 20950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:58.092167 20950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:58.093102 20950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:58.094038 20950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:58.102439 20958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:58.102468 20955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:58.102550 20950 server_base.cc:1034] running on GCE node
W20260929 01:00:58.102469 20956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:58.103315 20950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:58.104370 20950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:58.105568 20950 hybrid_clock.cc:648] HybridClock initialized: now 1790643658105553 us; error 28 us; skew 500 ppm
Sep 29 01:00:58 dist-test-slave-30s6 krb5kdc[20937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:58.108351 20950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:00:58.109205 20950 webserver.cc:460] Webserver started at http://127.0.0.1:45369/ using document root <none> and password file <none>
I20260929 01:00:58.109758 20950 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:58.109846 20950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:58.110097 20950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:58.112112 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "c116524150ac476a9c0645b0f99aeabb"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-30s6"
server_key: "1156c46f026efcd7f8ea6ee2d48ede33"
server_key_iv: "8abe72af9b7a049b12acba0130db8181"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.112674 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "c116524150ac476a9c0645b0f99aeabb"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-30s6"
server_key: "1156c46f026efcd7f8ea6ee2d48ede33"
server_key_iv: "8abe72af9b7a049b12acba0130db8181"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.116494 20950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:58.119009 20965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:58.120293 20950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:00:58.120427 20950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "c116524150ac476a9c0645b0f99aeabb"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-30s6"
server_key: "1156c46f026efcd7f8ea6ee2d48ede33"
server_key_iv: "8abe72af9b7a049b12acba0130db8181"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.120544 20950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:58.135373 20950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:58.138549 20950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:58.138756 20950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:58.146220 20950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37705
I20260929 01:00:58.146240 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37705 every 8 connection(s)
I20260929 01:00:58.147186 20950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:58.149878 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:58.150749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 20950
I20260929 01:00:58.150897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:58.151213 392 external_mini_cluster.cc:1428] Setting key 3b7cee452844d6fdd2c044c8fea4f419
I20260929 01:00:58.155968 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb: Bootstrap starting.
I20260929 01:00:58.158162 21018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:58.158941 21018 log.cc:824] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:00:58 dist-test-slave-30s6 krb5kdc[20937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:00:58.161077 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb: No bootstrap required, opened a new log
I20260929 01:00:58.165042 21018 raft_consensus.cc:348] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116524150ac476a9c0645b0f99aeabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37705 } }
I20260929 01:00:58.165251 21018 raft_consensus.cc:374] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:58.165290 21018 raft_consensus.cc:729] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c116524150ac476a9c0645b0f99aeabb, State: Initialized, Role: FOLLOWER
I20260929 01:00:58.165536 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:58.152665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35141 (local address 127.0.0.1:37705)
0929 01:00:58.153096 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:58.153110 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:58.153145 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:00:58.153778 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:58.153795 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:58.154129 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:58.154520 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:58.155285 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:58.156316 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:58.157171 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:58.157641 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:58.157825 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:58.160638 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:58.160643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:58.160655 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:58.160681 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:58.162443 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:58.162961 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:58.162965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:58.162969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:58.163077 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:58.163347 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:58.163350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:58.163351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:58.163727 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:58.163875 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:58.164238 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 01:00:58.164611 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":113,"threads_started":1}
I20260929 01:00:58.165817 21018 consensus_queue.cc:260] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116524150ac476a9c0645b0f99aeabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37705 } }
I20260929 01:00:58.165968 21018 raft_consensus.cc:388] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:58.166038 21018 raft_consensus.cc:482] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:58.166155 21018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:58.167075 21018 raft_consensus.cc:504] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116524150ac476a9c0645b0f99aeabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37705 } }
I20260929 01:00:58.167409 21018 leader_election.cc:302] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c116524150ac476a9c0645b0f99aeabb; no voters:
I20260929 01:00:58.167773 21018 leader_election.cc:288] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:58.167897 21023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:58.168154 21023 raft_consensus.cc:686] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [term 1 LEADER]: Becoming Leader. State: Replica: c116524150ac476a9c0645b0f99aeabb, State: Running, Role: LEADER
I20260929 01:00:58.168787 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:00:58.168927 21023 consensus_queue.cc:237] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116524150ac476a9c0645b0f99aeabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37705 } }
I20260929 01:00:58.168999 21018 sys_catalog.cc:564] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:58.170869 21024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c116524150ac476a9c0645b0f99aeabb. Latest consensus state: current_term: 1 leader_uuid: "c116524150ac476a9c0645b0f99aeabb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116524150ac476a9c0645b0f99aeabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37705 } } }
I20260929 01:00:58.171312 21024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:58.171171 21025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c116524150ac476a9c0645b0f99aeabb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116524150ac476a9c0645b0f99aeabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37705 } } }
I20260929 01:00:58.171838 21025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb [sys.catalog]: This master's current role is: LEADER
W20260929 01:00:58.174319 21042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:00:58.174433 21042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:00:58.174558 21043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:58.175547 21043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:58.180130 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:58.171309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34133 (local address 127.0.0.1:37705)
0929 01:00:58.171423 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:58.171428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:58.171644 (+ 216us) server_negotiation.cc:407] Connection header received
0929 01:00:58.171808 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:58.171811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:58.171852 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:58.171898 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:58.172811 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:58.173556 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:58.174729 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:58.174975 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:58.175033 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:58.176508 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:58.176512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:58.176514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:58.176538 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:58.178154 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:58.178821 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:58.178824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:58.178825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:58.178908 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:58.179258 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:58.179261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:58.179262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:58.179484 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:58.179612 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:58.179847 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 01:00:58.179958 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:00:58.181644 21043 catalog_manager.cc:1347] Generated new cluster ID: 967873d0b6d449b89dd6d5572c9e9494
I20260929 01:00:58.181742 21043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:58.190981 21043 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:58.192303 21043 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:58.200304 21043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c116524150ac476a9c0645b0f99aeabb: Generated new TSK 0
I20260929 01:00:58.201159 21043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:58.206588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 20950
2026-09-29T01:00:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:58 dist-test-slave-30s6 krb5kdc[21049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:58 dist-test-slave-30s6 krb5kdc[21049](info): set up 2 sockets
Sep 29 01:00:58 dist-test-slave-30s6 krb5kdc[21049](info): commencing operation
krb5kdc: starting...
W20260929 01:00:59.250439 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:00:59 dist-test-slave-30s6 krb5kdc[21049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:00:59.373095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:59.485107 21062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:59.485350 21062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:59.485390 21062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:59.489336 21062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:59.489389 21062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:59.489409 21062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:59.489426 21062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:59.489486 21062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:59.494637 21062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:00:59.495718 21062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:59.496771 21062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:00:59.505543 21062 server_base.cc:1034] running on GCE node
W20260929 01:00:59.505510 21067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:59.505487 21068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:59.505481 21070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:59.506417 21062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:59.507479 21062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:59.508638 21062 hybrid_clock.cc:648] HybridClock initialized: now 1790643659508610 us; error 42 us; skew 500 ppm
Sep 29 01:00:59 dist-test-slave-30s6 krb5kdc[21049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:59.511518 21062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:00:59.512451 21062 webserver.cc:460] Webserver started at http://127.0.0.1:40203/ using document root <none> and password file <none>
I20260929 01:00:59.513002 21062 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:59.513063 21062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:59.513273 21062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:59.515327 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "419967174c0b4446bf813f37ce38b147"
format_stamp: "Formatted at 2026-09-29 01:00:59 on dist-test-slave-30s6"
server_key: "5217ea5fafcba4a305772e2b0ac5349e"
server_key_iv: "054fb16b61c1c50d4857d49f2dabf130"
server_key_version: "encryptionkey@0"
I20260929 01:00:59.515887 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "419967174c0b4446bf813f37ce38b147"
format_stamp: "Formatted at 2026-09-29 01:00:59 on dist-test-slave-30s6"
server_key: "5217ea5fafcba4a305772e2b0ac5349e"
server_key_iv: "054fb16b61c1c50d4857d49f2dabf130"
server_key_version: "encryptionkey@0"
I20260929 01:00:59.519549 21062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:59.521920 21077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:59.523231 21062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:59.523370 21062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "419967174c0b4446bf813f37ce38b147"
format_stamp: "Formatted at 2026-09-29 01:00:59 on dist-test-slave-30s6"
server_key: "5217ea5fafcba4a305772e2b0ac5349e"
server_key_iv: "054fb16b61c1c50d4857d49f2dabf130"
server_key_version: "encryptionkey@0"
I20260929 01:00:59.523475 21062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:59.540126 21062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:59.543581 21062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:59.543790 21062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:59.551216 21062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43425
I20260929 01:00:59.551232 21129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43425 every 8 connection(s)
I20260929 01:00:59.552203 21062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:59.554988 21130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:59.559783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21062
I20260929 01:00:59.559937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:00:59.560241 392 external_mini_cluster.cc:1428] Setting key 783dc07585e18e892f5d040120ef1eb4
I20260929 01:00:59.560812 21130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147: Bootstrap starting.
I20260929 01:00:59.563176 21130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:59.564036 21130 log.cc:824] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:59.566102 21130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147: No bootstrap required, opened a new log
I20260929 01:00:59.568918 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.561569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41463 (local address 127.0.0.1:43425)
0929 01:00:59.561988 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.562003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.562037 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:00:59.562709 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.562725 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.563053 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.563399 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.563724 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:59.564740 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:59.565884 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:59.566426 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:59.566542 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:59.567334 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.567339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.567351 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.567709 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.567758 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.567778 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.567852 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.568013 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.568261 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
I20260929 01:00:59.569918 21130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419967174c0b4446bf813f37ce38b147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260929 01:00:59.570130 21130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:59.570190 21130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419967174c0b4446bf813f37ce38b147, State: Initialized, Role: FOLLOWER
I20260929 01:00:59.570627 21130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419967174c0b4446bf813f37ce38b147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260929 01:00:59.570777 21130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:59.570886 21130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:59.571002 21130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:59.571830 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:00:59.571992 21130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419967174c0b4446bf813f37ce38b147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260929 01:00:59.572338 21130 leader_election.cc:302] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419967174c0b4446bf813f37ce38b147; no voters:
I20260929 01:00:59.572669 21130 leader_election.cc:288] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:59.572768 21135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:59.572988 21135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [term 1 LEADER]: Becoming Leader. State: Replica: 419967174c0b4446bf813f37ce38b147, State: Running, Role: LEADER
I20260929 01:00:59.573333 21135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419967174c0b4446bf813f37ce38b147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260929 01:00:59.573977 21130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:59.574915 21137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 419967174c0b4446bf813f37ce38b147. Latest consensus state: current_term: 1 leader_uuid: "419967174c0b4446bf813f37ce38b147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419967174c0b4446bf813f37ce38b147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } } }
I20260929 01:00:59.575047 21137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:59.575363 21136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "419967174c0b4446bf813f37ce38b147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419967174c0b4446bf813f37ce38b147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } } }
I20260929 01:00:59.575452 21136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:59.575860 21149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:59.579212 21149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:59.579916 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.574673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54085 (local address 127.0.0.1:43425)
0929 01:00:59.574808 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.574814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.575398 (+ 584us) server_negotiation.cc:407] Connection header received
0929 01:00:59.575590 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.575594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.575646 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.575717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.576146 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:59.576887 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:59.578058 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:59.578427 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:59.578540 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:59.579437 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.579441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.579444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.579458 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.579488 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.579501 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.579580 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.579717 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.579775 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:00:59.584992 21149 catalog_manager.cc:1347] Generated new cluster ID: 30819f0bd80643f18081d2b2a2fdb738
I20260929 01:00:59.585057 21149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:59.603441 21149 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:59.604756 21149 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:59.609736 21149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 419967174c0b4446bf813f37ce38b147: Generated new TSK 0
I20260929 01:00:59.610494 21149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:59.616922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21062
2026-09-29T01:00:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:59 dist-test-slave-30s6 krb5kdc[21161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:59 dist-test-slave-30s6 krb5kdc[21161](info): set up 2 sockets
Sep 29 01:00:59 dist-test-slave-30s6 krb5kdc[21161](info): commencing operation
krb5kdc: starting...
W20260929 01:01:00.670418 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:00 dist-test-slave-30s6 krb5kdc[21161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:01:00.793217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:00.903668 21174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:00.904026 21174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:00.904120 21174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:00.908275 21174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:00.908383 21174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:00.908422 21174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:00.908460 21174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:00.908490 21174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:00.913973 21174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:00.915102 21174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:00.916178 21174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:00.924618 21180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:00.924662 21179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:00.924777 21174 server_base.cc:1034] running on GCE node
W20260929 01:01:00.924957 21182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:00.925467 21174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:00.926556 21174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:00.927717 21174 hybrid_clock.cc:648] HybridClock initialized: now 1790643660927705 us; error 32 us; skew 500 ppm
Sep 29 01:01:00 dist-test-slave-30s6 krb5kdc[21161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:00.930560 21174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:01:00.931355 21174 webserver.cc:460] Webserver started at http://127.0.0.1:45557/ using document root <none> and password file <none>
I20260929 01:01:00.931854 21174 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:00.931911 21174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:00.932070 21174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:00.934027 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "2d479e9471ee42699b089ca6678b950d"
format_stamp: "Formatted at 2026-09-29 01:01:00 on dist-test-slave-30s6"
server_key: "a0e17ecb055c0c1e87f46e3f15ea19c5"
server_key_iv: "0f0434eb884f84c31af28666293dbcc8"
server_key_version: "encryptionkey@0"
I20260929 01:01:00.934726 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d479e9471ee42699b089ca6678b950d"
format_stamp: "Formatted at 2026-09-29 01:01:00 on dist-test-slave-30s6"
server_key: "a0e17ecb055c0c1e87f46e3f15ea19c5"
server_key_iv: "0f0434eb884f84c31af28666293dbcc8"
server_key_version: "encryptionkey@0"
I20260929 01:01:00.938663 21174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:00.941387 21189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:00.942816 21174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:00.942970 21174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "2d479e9471ee42699b089ca6678b950d"
format_stamp: "Formatted at 2026-09-29 01:01:00 on dist-test-slave-30s6"
server_key: "a0e17ecb055c0c1e87f46e3f15ea19c5"
server_key_iv: "0f0434eb884f84c31af28666293dbcc8"
server_key_version: "encryptionkey@0"
I20260929 01:01:00.943096 21174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:00.947911 21174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:00.951319 21174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:00.951509 21174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:00.958451 21174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32779
I20260929 01:01:00.958474 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32779 every 8 connection(s)
I20260929 01:01:00.959422 21174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:00.962007 21242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:00.967314 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d: Bootstrap starting.
I20260929 01:01:00.969465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21174
I20260929 01:01:00.969617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:00.969691 21242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:00.969936 392 external_mini_cluster.cc:1428] Setting key 8acb54e12f762634adde44153fc033ef
I20260929 01:01:00.970620 21242 log.cc:824] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:00.972509 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d: No bootstrap required, opened a new log
I20260929 01:01:00.974948 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.971106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59829 (local address 127.0.0.1:32779)
0929 01:01:00.971551 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:00.971578 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:00.971613 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:01:00.972246 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:00.972262 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:00.972534 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:00.972877 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:00.973707 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:00.973711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:00.973721 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:00.973731 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:00.973785 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:00.973804 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:00.973890 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:00.974020 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 01:01:00.974189 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":144,"threads_started":1}
I20260929 01:01:00.976485 21242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d479e9471ee42699b089ca6678b950d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } }
I20260929 01:01:00.976791 21242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:00.976861 21242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d479e9471ee42699b089ca6678b950d, State: Initialized, Role: FOLLOWER
I20260929 01:01:00.977314 21242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d479e9471ee42699b089ca6678b950d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } }
I20260929 01:01:00.977473 21242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:00.977571 21242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:00.977696 21242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:00.977774 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:00.978780 21242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d479e9471ee42699b089ca6678b950d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } }
I20260929 01:01:00.979187 21242 leader_election.cc:302] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d479e9471ee42699b089ca6678b950d; no voters:
I20260929 01:01:00.979540 21242 leader_election.cc:288] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:00.979610 21251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:00.979847 21251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [term 1 LEADER]: Becoming Leader. State: Replica: 2d479e9471ee42699b089ca6678b950d, State: Running, Role: LEADER
I20260929 01:01:00.980154 21251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d479e9471ee42699b089ca6678b950d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } }
I20260929 01:01:00.980901 21242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:00.981339 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d479e9471ee42699b089ca6678b950d. Latest consensus state: current_term: 1 leader_uuid: "2d479e9471ee42699b089ca6678b950d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d479e9471ee42699b089ca6678b950d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } } }
I20260929 01:01:00.981457 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:00.981660 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d479e9471ee42699b089ca6678b950d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d479e9471ee42699b089ca6678b950d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } } }
I20260929 01:01:00.981758 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:00.982036 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.979946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35009 (local address 127.0.0.1:32779)
0929 01:01:00.980121 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:00.980126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:00.980365 (+ 239us) server_negotiation.cc:407] Connection header received
0929 01:01:00.980547 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:00.980550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:00.980586 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:00.980676 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:00.981458 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:00.981464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:00.981469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:00.981483 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:00.981511 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:00.981523 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:00.981592 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:00.981859 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 01:01:00.981903 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260929 01:01:00.985497 21267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:00.985566 21267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:00.985648 21256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:00.986490 21256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:00.992328 21256 catalog_manager.cc:1347] Generated new cluster ID: 2d57fd0ba17843b9ae26ca38cf0ab263
I20260929 01:01:00.992404 21256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:01.011480 21256 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:01.011862 21256 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:01.017695 21256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d479e9471ee42699b089ca6678b950d: Generated new TSK 0
I20260929 01:01:01.018523 21256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:01.029515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21174
2026-09-29T01:01:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:01 dist-test-slave-30s6 krb5kdc[21273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:01 dist-test-slave-30s6 krb5kdc[21273](info): set up 2 sockets
Sep 29 01:01:01 dist-test-slave-30s6 krb5kdc[21273](info): commencing operation
krb5kdc: starting...
W20260929 01:01:02.066604 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:02 dist-test-slave-30s6 krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:01:02.194646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:02.307195 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:02.307495 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:02.307539 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:02.311630 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:02.311697 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:02.311719 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:02.311736 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:02.311751 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:02.317200 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:02.318311 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:02.319296 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:02.327865 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:02.327950 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:02.328023 21286 server_base.cc:1034] running on GCE node
W20260929 01:01:02.327875 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:02.328756 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:02.329877 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:02.331050 21286 hybrid_clock.cc:648] HybridClock initialized: now 1790643662331036 us; error 53 us; skew 500 ppm
Sep 29 01:01:02 dist-test-slave-30s6 krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:02.339646 21286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:01:02.340700 21286 webserver.cc:460] Webserver started at http://127.0.0.1:46151/ using document root <none> and password file <none>
I20260929 01:01:02.341326 21286 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:02.341423 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:02.341663 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:02.343922 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b"
format_stamp: "Formatted at 2026-09-29 01:01:02 on dist-test-slave-30s6"
server_key: "57a6e22bcea9813400ce2f991639fc9a"
server_key_iv: "b9b59e643c07bb51a3041ae759a01847"
server_key_version: "encryptionkey@0"
I20260929 01:01:02.344501 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b"
format_stamp: "Formatted at 2026-09-29 01:01:02 on dist-test-slave-30s6"
server_key: "57a6e22bcea9813400ce2f991639fc9a"
server_key_iv: "b9b59e643c07bb51a3041ae759a01847"
server_key_version: "encryptionkey@0"
I20260929 01:01:02.348711 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:01:02.351504 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:02.352806 21286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:01:02.352964 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b"
format_stamp: "Formatted at 2026-09-29 01:01:02 on dist-test-slave-30s6"
server_key: "57a6e22bcea9813400ce2f991639fc9a"
server_key_iv: "b9b59e643c07bb51a3041ae759a01847"
server_key_version: "encryptionkey@0"
I20260929 01:01:02.353075 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:02.371534 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:02.375136 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:02.375355 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:02.382905 21286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38057
I20260929 01:01:02.382925 21353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38057 every 8 connection(s)
I20260929 01:01:02.383904 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:02.386644 21354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:02.391698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21286
I20260929 01:01:02.391848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:02.392140 392 external_mini_cluster.cc:1428] Setting key 7d8cc801e483ab1e2ae405b33c13d6b0
I20260929 01:01:02.392204 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b: Bootstrap starting.
I20260929 01:01:02.394701 21354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:02.395421 21354 log.cc:824] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:02.397220 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b: No bootstrap required, opened a new log
I20260929 01:01:02.400633 21354 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } }
I20260929 01:01:02.400878 21354 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:02.400957 21354 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0f9f601e874abc9b2b7c2d53c32e4b, State: Initialized, Role: FOLLOWER
Sep 29 01:01:02 dist-test-slave-30s6 krb5kdc[21273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:01:02.401470 21354 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } }
I20260929 01:01:02.401661 21354 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:02.401736 21354 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:02.401853 21354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:02.403067 21354 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } }
I20260929 01:01:02.403439 21354 leader_election.cc:302] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba0f9f601e874abc9b2b7c2d53c32e4b; no voters:
I20260929 01:01:02.403775 21354 leader_election.cc:288] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:02.403905 21359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:02.404155 21359 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [term 1 LEADER]: Becoming Leader. State: Replica: ba0f9f601e874abc9b2b7c2d53c32e4b, State: Running, Role: LEADER
I20260929 01:01:02.404565 21359 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } }
I20260929 01:01:02.405148 21354 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:02.408836 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:02.393556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:41183 (local address 127.0.0.1:38057)
0929 01:01:02.393987 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:02.394003 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:02.394050 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:01:02.394780 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:02.394802 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:02.395166 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:02.395503 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:02.396392 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.397435 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.398369 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.398855 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.399049 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:02.401973 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:02.401981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:02.401999 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:02.402095 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:02.404415 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:02.405058 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:02.405065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:02.405071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:02.405223 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:02.405615 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:02.405618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:02.405620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:02.406240 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:02.406480 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:02.407493 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0929 01:01:02.407760 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":120,"threads_started":1}
I20260929 01:01:02.409536 21361 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba0f9f601e874abc9b2b7c2d53c32e4b. Latest consensus state: current_term: 1 leader_uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } } }
I20260929 01:01:02.409528 21360 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f9f601e874abc9b2b7c2d53c32e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } } }
I20260929 01:01:02.410082 21360 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:02.410332 21361 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b [sys.catalog]: This master's current role is: LEADER
W20260929 01:01:02.411863 21373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:02.411962 21373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:02.412108 21376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:02.412997 21376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:02.413523 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:02.419158 21376 catalog_manager.cc:1347] Generated new cluster ID: 52c0ace8ce55436988191533f170c542
I20260929 01:01:02.419241 21376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:02.426090 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:02.417940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:37699 (local address 127.0.0.1:38057)
0929 01:01:02.418054 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:02.418059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:02.418261 (+ 202us) server_negotiation.cc:407] Connection header received
0929 01:01:02.418414 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:02.418418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:02.418476 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:02.418569 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:02.419438 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.419963 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.420796 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.421060 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.421174 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:02.422770 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:02.422774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:02.422776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:02.422814 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:02.424296 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:02.424788 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:02.424793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:02.424800 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:02.424873 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:02.425204 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:02.425207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:02.425209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:02.425383 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:02.425556 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:02.425823 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 01:01:02.425917 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 01:01:02.429281 21376 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:02.430299 21376 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:02.437348 21376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba0f9f601e874abc9b2b7c2d53c32e4b: Generated new TSK 0
I20260929 01:01:02.438102 21376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:02.442082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21286
W20260929 01:01:02.447381 21271 connection.cc:535] client connection to 127.0.0.1:38057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38057): Connection reset by peer (error 104)
2026-09-29T01:01:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:02 dist-test-slave-30s6 krb5kdc[21385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:02 dist-test-slave-30s6 krb5kdc[21385](info): set up 2 sockets
Sep 29 01:01:02 dist-test-slave-30s6 krb5kdc[21385](info): commencing operation
krb5kdc: starting...
W20260929 01:01:03.486521 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:03 dist-test-slave-30s6 krb5kdc[21385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:01:03.613042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:03.724246 21398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:03.724527 21398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:03.724571 21398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:03.728622 21398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:03.728684 21398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:03.728704 21398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:03.728721 21398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:03.728736 21398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:03.734048 21398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:03.735179 21398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:03.736243 21398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:03.745056 21404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:03.745033 21406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:03.745046 21403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:03.745343 21398 server_base.cc:1034] running on GCE node
I20260929 01:01:03.745919 21398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:03.747134 21398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:03.748301 21398 hybrid_clock.cc:648] HybridClock initialized: now 1790643663748289 us; error 30 us; skew 500 ppm
Sep 29 01:01:03 dist-test-slave-30s6 krb5kdc[21385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:03.751085 21398 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:01:03.751991 21398 webserver.cc:460] Webserver started at http://127.0.0.1:38489/ using document root <none> and password file <none>
I20260929 01:01:03.752511 21398 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:03.752568 21398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:03.752727 21398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:03.754839 21398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "ac80b80264f24d51973f8440f620ae39"
format_stamp: "Formatted at 2026-09-29 01:01:03 on dist-test-slave-30s6"
server_key: "b47202d74314dd80f081e1db3c7d56d9"
server_key_iv: "40e188e695c8db4b89f66a8c09619361"
server_key_version: "encryptionkey@0"
I20260929 01:01:03.755357 21398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac80b80264f24d51973f8440f620ae39"
format_stamp: "Formatted at 2026-09-29 01:01:03 on dist-test-slave-30s6"
server_key: "b47202d74314dd80f081e1db3c7d56d9"
server_key_iv: "40e188e695c8db4b89f66a8c09619361"
server_key_version: "encryptionkey@0"
I20260929 01:01:03.758968 21398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:01:03.761266 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:03.762666 21398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:01:03.762763 21398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "ac80b80264f24d51973f8440f620ae39"
format_stamp: "Formatted at 2026-09-29 01:01:03 on dist-test-slave-30s6"
server_key: "b47202d74314dd80f081e1db3c7d56d9"
server_key_iv: "40e188e695c8db4b89f66a8c09619361"
server_key_version: "encryptionkey@0"
I20260929 01:01:03.762863 21398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:03.782095 21398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:03.785104 21398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:03.785257 21398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:03.791999 21398 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45185
I20260929 01:01:03.792071 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45185 every 8 connection(s)
I20260929 01:01:03.792969 21398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:03.795889 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:03.799687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21398
I20260929 01:01:03.799851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:03.800127 392 external_mini_cluster.cc:1428] Setting key 9e5828fd693ef7aadaabcbf116577cf3
I20260929 01:01:03.801925 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39: Bootstrap starting.
I20260929 01:01:03.804280 21466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:03.805063 21466 log.cc:824] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:03.807168 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39: No bootstrap required, opened a new log
I20260929 01:01:03.809402 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:03.801556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:55649 (local address 127.0.0.1:45185)
0929 01:01:03.802004 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:03.802018 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:03.802053 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:01:03.802781 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:03.802798 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:03.803138 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:03.803487 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:03.803911 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.804946 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.805869 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.806412 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.806594 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:03.807573 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:03.807578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:03.807591 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:03.807968 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:03.808020 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:03.808042 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:03.808182 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:03.808370 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 01:01:03.808503 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260929 01:01:03.810520 21466 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac80b80264f24d51973f8440f620ae39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45185 } }
I20260929 01:01:03.810808 21466 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:03.810882 21466 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac80b80264f24d51973f8440f620ae39, State: Initialized, Role: FOLLOWER
I20260929 01:01:03.811376 21466 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac80b80264f24d51973f8440f620ae39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45185 } }
I20260929 01:01:03.811553 21466 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:03.811652 21466 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:03.811766 21466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:03.812528 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:03.812927 21466 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac80b80264f24d51973f8440f620ae39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45185 } }
I20260929 01:01:03.813280 21466 leader_election.cc:302] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac80b80264f24d51973f8440f620ae39; no voters:
I20260929 01:01:03.813638 21466 leader_election.cc:288] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:03.813722 21471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:03.813985 21471 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [term 1 LEADER]: Becoming Leader. State: Replica: ac80b80264f24d51973f8440f620ae39, State: Running, Role: LEADER
I20260929 01:01:03.814373 21471 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac80b80264f24d51973f8440f620ae39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45185 } }
I20260929 01:01:03.815016 21466 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:03.815598 21471 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac80b80264f24d51973f8440f620ae39. Latest consensus state: current_term: 1 leader_uuid: "ac80b80264f24d51973f8440f620ae39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac80b80264f24d51973f8440f620ae39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45185 } } }
I20260929 01:01:03.815725 21471 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:03.816097 21474 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac80b80264f24d51973f8440f620ae39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac80b80264f24d51973f8440f620ae39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45185 } } }
I20260929 01:01:03.816260 21474 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:03.817425 21481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:03.820225 21481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:03.825985 21481 catalog_manager.cc:1347] Generated new cluster ID: 02ce9877241f4fd1940994cadc0329c6
I20260929 01:01:03.826053 21481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:03.826750 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:03.814696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:36389 (local address 127.0.0.1:45185)
0929 01:01:03.814814 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:03.814818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:03.822161 (+ 7343us) server_negotiation.cc:407] Connection header received
0929 01:01:03.822371 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:03.822374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:03.822431 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:03.822533 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:03.822854 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.823393 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.824509 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.824866 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.824988 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:03.826033 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:03.826037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:03.826040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:03.826053 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:03.826082 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:03.826095 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:03.826522 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:03.826578 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 01:01:03.826613 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:01:03.840513 21481 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:03.841778 21481 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:03.863139 21481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac80b80264f24d51973f8440f620ae39: Generated new TSK 0
I20260929 01:01:03.863945 21481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:03.877820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21398
W20260929 01:01:03.884152 21383 connection.cc:535] client connection to 127.0.0.1:45185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45185): Connection reset by peer (error 104)
2026-09-29T01:01:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:03 dist-test-slave-30s6 krb5kdc[21497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:03 dist-test-slave-30s6 krb5kdc[21497](info): set up 2 sockets
Sep 29 01:01:03 dist-test-slave-30s6 krb5kdc[21497](info): commencing operation
krb5kdc: starting...
W20260929 01:01:04.922519 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:05 dist-test-slave-30s6 krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:01:05.055475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:05.172406 21510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:05.172708 21510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:05.172753 21510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:05.177014 21510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:05.177098 21510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:05.177117 21510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:05.177134 21510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:05.177148 21510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:05.182914 21510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:05.184122 21510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:05.185268 21510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:05.194766 21516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:05.194977 21515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:05.194906 21518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:05.194784 21510 server_base.cc:1034] running on GCE node
I20260929 01:01:05.195677 21510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:05.196853 21510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:05.198035 21510 hybrid_clock.cc:648] HybridClock initialized: now 1790643665198023 us; error 34 us; skew 500 ppm
Sep 29 01:01:05 dist-test-slave-30s6 krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:05.201225 21510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:01:05.202301 21510 webserver.cc:460] Webserver started at http://127.0.0.1:35523/ using document root <none> and password file <none>
I20260929 01:01:05.202929 21510 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:05.203027 21510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:05.203282 21510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:05.205726 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "3163be883c38433ba47b1f7b379600d3"
format_stamp: "Formatted at 2026-09-29 01:01:05 on dist-test-slave-30s6"
server_key: "eeac21bb1b27354704ea95699e1124bf"
server_key_iv: "2b6b99dd2896486b642fa9815c12ec6f"
server_key_version: "encryptionkey@0"
I20260929 01:01:05.206445 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "3163be883c38433ba47b1f7b379600d3"
format_stamp: "Formatted at 2026-09-29 01:01:05 on dist-test-slave-30s6"
server_key: "eeac21bb1b27354704ea95699e1124bf"
server_key_iv: "2b6b99dd2896486b642fa9815c12ec6f"
server_key_version: "encryptionkey@0"
I20260929 01:01:05.210453 21510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:01:05.213157 21525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:05.214774 21510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:05.214955 21510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "3163be883c38433ba47b1f7b379600d3"
format_stamp: "Formatted at 2026-09-29 01:01:05 on dist-test-slave-30s6"
server_key: "eeac21bb1b27354704ea95699e1124bf"
server_key_iv: "2b6b99dd2896486b642fa9815c12ec6f"
server_key_version: "encryptionkey@0"
I20260929 01:01:05.215101 21510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:05.220323 21510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:05.224018 21510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:05.224246 21510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:05.231804 21510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41347
I20260929 01:01:05.231837 21577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41347 every 8 connection(s)
I20260929 01:01:05.232884 21510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:05.235962 21578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:05.241768 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3: Bootstrap starting.
I20260929 01:01:05.242554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21510
I20260929 01:01:05.242728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:05.243101 392 external_mini_cluster.cc:1428] Setting key c4860b91310d1f6d2ec0bf43b43b0e95
I20260929 01:01:05.244829 21578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:05.245625 21578 log.cc:824] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:05.248081 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3: No bootstrap required, opened a new log
I20260929 01:01:05.248739 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:05.244369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:44031 (local address 127.0.0.1:41347)
0929 01:01:05.244870 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:05.244885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:05.244921 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:01:05.245658 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:05.245679 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:05.246070 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:05.246495 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:05.247382 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:05.247386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:05.247399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:05.247410 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:05.247467 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:05.247486 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:05.247577 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:05.247727 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 01:01:05.247915 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":178,"threads_started":1}
I20260929 01:01:05.251897 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:05.252431 21578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163be883c38433ba47b1f7b379600d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } }
I20260929 01:01:05.252734 21578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:05.252799 21578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3163be883c38433ba47b1f7b379600d3, State: Initialized, Role: FOLLOWER
I20260929 01:01:05.253412 21578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163be883c38433ba47b1f7b379600d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } }
I20260929 01:01:05.253683 21578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:05.253796 21578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:05.253927 21578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:05.255264 21578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163be883c38433ba47b1f7b379600d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } }
I20260929 01:01:05.255635 21578 leader_election.cc:302] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3163be883c38433ba47b1f7b379600d3; no voters:
I20260929 01:01:05.255977 21578 leader_election.cc:288] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:05.256119 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:05.256394 21588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3163be883c38433ba47b1f7b379600d3, State: Running, Role: LEADER
I20260929 01:01:05.256608 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:05.254498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:39409 (local address 127.0.0.1:41347)
0929 01:01:05.254626 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:05.254631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:05.254824 (+ 193us) server_negotiation.cc:407] Connection header received
0929 01:01:05.254925 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:05.254931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:05.254971 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:05.255069 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:05.256092 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:05.256095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:05.256098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:05.256110 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:05.256140 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:05.256153 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:05.256247 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:05.256438 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 01:01:05.256484 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:01:05.256923 21588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163be883c38433ba47b1f7b379600d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } }
I20260929 01:01:05.257506 21578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:05.258934 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3163be883c38433ba47b1f7b379600d3. Latest consensus state: current_term: 1 leader_uuid: "3163be883c38433ba47b1f7b379600d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163be883c38433ba47b1f7b379600d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } } }
I20260929 01:01:05.259085 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:05.259495 21600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:05.259451 21589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3163be883c38433ba47b1f7b379600d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163be883c38433ba47b1f7b379600d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } } }
I20260929 01:01:05.259541 21589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:05.263314 21600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:05.269309 21600 catalog_manager.cc:1347] Generated new cluster ID: fe0a9f62c7a14b738b51ae0e3aaf7c28
I20260929 01:01:05.269397 21600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:05.284803 21600 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:05.285190 21600 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:05.292234 21600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3163be883c38433ba47b1f7b379600d3: Generated new TSK 0
I20260929 01:01:05.293022 21600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:05.302024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21510
2026-09-29T01:01:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:05 dist-test-slave-30s6 krb5kdc[21609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:05 dist-test-slave-30s6 krb5kdc[21609](info): set up 2 sockets
Sep 29 01:01:05 dist-test-slave-30s6 krb5kdc[21609](info): commencing operation
krb5kdc: starting...
W20260929 01:01:06.354465 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:06 dist-test-slave-30s6 krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:01:06.476533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:06.583395 21622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:06.583675 21622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:06.583752 21622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:06.587736 21622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:06.587810 21622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:06.587857 21622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:06.587896 21622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:06.587930 21622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:06.593043 21622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:06.594018 21622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:06.595042 21622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:06.603360 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:06.603466 21622 server_base.cc:1034] running on GCE node
W20260929 01:01:06.603400 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:06.603375 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:06.604135 21622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:06.605204 21622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:06.606389 21622 hybrid_clock.cc:648] HybridClock initialized: now 1790643666606378 us; error 26 us; skew 500 ppm
Sep 29 01:01:06 dist-test-slave-30s6 krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:06.609069 21622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:01:06.609906 21622 webserver.cc:460] Webserver started at http://127.0.0.1:35163/ using document root <none> and password file <none>
I20260929 01:01:06.610517 21622 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:06.610608 21622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:06.610836 21622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:06.612831 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "45a0a920c0954ba3ab6b12420068b10e"
format_stamp: "Formatted at 2026-09-29 01:01:06 on dist-test-slave-30s6"
server_key: "52afcab5215c791de97961319f24aaba"
server_key_iv: "883f220165f6ef0950d0e908ea8068ad"
server_key_version: "encryptionkey@0"
I20260929 01:01:06.613384 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "45a0a920c0954ba3ab6b12420068b10e"
format_stamp: "Formatted at 2026-09-29 01:01:06 on dist-test-slave-30s6"
server_key: "52afcab5215c791de97961319f24aaba"
server_key_iv: "883f220165f6ef0950d0e908ea8068ad"
server_key_version: "encryptionkey@0"
I20260929 01:01:06.616926 21622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260929 01:01:06.619320 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:06.620546 21622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:01:06.620682 21622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "45a0a920c0954ba3ab6b12420068b10e"
format_stamp: "Formatted at 2026-09-29 01:01:06 on dist-test-slave-30s6"
server_key: "52afcab5215c791de97961319f24aaba"
server_key_iv: "883f220165f6ef0950d0e908ea8068ad"
server_key_version: "encryptionkey@0"
I20260929 01:01:06.620792 21622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:06.634724 21622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:06.637362 21622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:06.637552 21622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:06.644369 21622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38753
I20260929 01:01:06.644400 21689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38753 every 8 connection(s)
I20260929 01:01:06.645259 21622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:06.647897 21690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:06.652081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21622
I20260929 01:01:06.652237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:06.652509 392 external_mini_cluster.cc:1428] Setting key 7885e09f0b765337c3534b1bb50e8090
I20260929 01:01:06.653688 21690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e: Bootstrap starting.
I20260929 01:01:06.656376 21690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:06.657227 21690 log.cc:824] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:06.659241 21690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e: No bootstrap required, opened a new log
Sep 29 01:01:06 dist-test-slave-30s6 krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:01:06.663304 21690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a0a920c0954ba3ab6b12420068b10e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38753 } }
I20260929 01:01:06.663522 21690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:06.663620 21690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a0a920c0954ba3ab6b12420068b10e, State: Initialized, Role: FOLLOWER
I20260929 01:01:06.664062 21690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a0a920c0954ba3ab6b12420068b10e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38753 } }
I20260929 01:01:06.664184 21690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:06.664278 21690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:06.664407 21690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:06.665512 21690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a0a920c0954ba3ab6b12420068b10e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38753 } }
I20260929 01:01:06.665874 21690 leader_election.cc:302] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45a0a920c0954ba3ab6b12420068b10e; no voters:
I20260929 01:01:06.666213 21690 leader_election.cc:288] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:06.666283 21695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:06.666488 21695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [term 1 LEADER]: Becoming Leader. State: Replica: 45a0a920c0954ba3ab6b12420068b10e, State: Running, Role: LEADER
I20260929 01:01:06.666759 21695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a0a920c0954ba3ab6b12420068b10e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38753 } }
I20260929 01:01:06.667472 21690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:06.667886 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:06.653941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:60393 (local address 127.0.0.1:38753)
0929 01:01:06.654362 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:06.654380 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:06.654421 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:01:06.655260 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:06.655281 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:06.655698 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:06.656075 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:06.656771 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:06.658142 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:06.659018 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:06.659678 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:06.659854 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:06.662698 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:06.662703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:06.662716 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:06.662742 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:06.664738 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:06.665277 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:06.665282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:06.665286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:06.665402 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:06.665753 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:06.665755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:06.665757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:06.666248 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:06.666472 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:06.666782 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 01:01:06.667017 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260929 01:01:06.667896 21695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45a0a920c0954ba3ab6b12420068b10e. Latest consensus state: current_term: 1 leader_uuid: "45a0a920c0954ba3ab6b12420068b10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a0a920c0954ba3ab6b12420068b10e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38753 } } }
I20260929 01:01:06.668342 21695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:06.668594 21695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45a0a920c0954ba3ab6b12420068b10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a0a920c0954ba3ab6b12420068b10e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38753 } } }
I20260929 01:01:06.668694 21695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:06.668996 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:06.671514 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:06.672212 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:06.681653 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:06.673849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:43035 (local address 127.0.0.1:38753)
0929 01:01:06.674092 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:06.674096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:06.674115 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:01:06.674141 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:06.674143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:06.674219 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:06.674295 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:06.674976 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:06.675488 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:06.676294 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:06.676554 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:06.676656 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:06.678009 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:06.678012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:06.678015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:06.678038 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:06.679482 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:06.679942 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:06.679948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:06.679949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:06.680025 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:06.680347 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:06.680350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:06.680352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:06.680514 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:06.681068 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:06.681390 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 01:01:06.681481 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260929 01:01:06.683068 21704 catalog_manager.cc:1347] Generated new cluster ID: e35add8e98524545b65f8dd1b2754bbf
I20260929 01:01:06.683128 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:06.688711 21704 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:06.689589 21704 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:06.700070 21704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45a0a920c0954ba3ab6b12420068b10e: Generated new TSK 0
I20260929 01:01:06.700747 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:06.708992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21622
W20260929 01:01:06.714052 21607 connection.cc:535] client connection to 127.0.0.1:38753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38753): Connection reset by peer (error 104)
2026-09-29T01:01:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:06 dist-test-slave-30s6 krb5kdc[21721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:06 dist-test-slave-30s6 krb5kdc[21721](info): set up 2 sockets
Sep 29 01:01:06 dist-test-slave-30s6 krb5kdc[21721](info): commencing operation
krb5kdc: starting...
W20260929 01:01:07.750473 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:07 dist-test-slave-30s6 krb5kdc[21721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:01:07.871294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:07.979532 21734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:07.979806 21734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:07.979848 21734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:07.983989 21734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:07.984053 21734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:07.984078 21734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:07.984095 21734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:07.984135 21734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:07.989440 21734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:07.990545 21734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:07.991565 21734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:08.000142 21742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:08.000134 21739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:08.000155 21740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:08.000360 21734 server_base.cc:1034] running on GCE node
I20260929 01:01:08.001029 21734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:08.002100 21734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:08.003320 21734 hybrid_clock.cc:648] HybridClock initialized: now 1790643668003289 us; error 43 us; skew 500 ppm
Sep 29 01:01:08 dist-test-slave-30s6 krb5kdc[21721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:08.009681 21734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:01:08.010691 21734 webserver.cc:460] Webserver started at http://127.0.0.1:41305/ using document root <none> and password file <none>
I20260929 01:01:08.011243 21734 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:08.011363 21734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:08.011590 21734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:08.013602 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "e02cab604c4746aeae579398a42e5010"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-30s6"
server_key: "2f0087da2265c0645cdbcb34f12fa28a"
server_key_iv: "d20465c658f2181783df870f3ac1c3f4"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.014158 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "e02cab604c4746aeae579398a42e5010"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-30s6"
server_key: "2f0087da2265c0645cdbcb34f12fa28a"
server_key_iv: "d20465c658f2181783df870f3ac1c3f4"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.017954 21734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:08.020439 21749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:08.021708 21734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:01:08.021840 21734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "e02cab604c4746aeae579398a42e5010"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-30s6"
server_key: "2f0087da2265c0645cdbcb34f12fa28a"
server_key_iv: "d20465c658f2181783df870f3ac1c3f4"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.021951 21734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:08.038019 21734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:08.041482 21734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:08.041687 21734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:08.049118 21734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41793
I20260929 01:01:08.049140 21801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41793 every 8 connection(s)
I20260929 01:01:08.050066 21734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:08.052783 21802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:08.057367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21734
I20260929 01:01:08.057533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:08.057827 392 external_mini_cluster.cc:1428] Setting key 052aadf0084fea4e76f1e11edb0588a0
I20260929 01:01:08.058701 21802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010: Bootstrap starting.
I20260929 01:01:08.061420 21802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:08.062323 21802 log.cc:824] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:08.064469 21802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010: No bootstrap required, opened a new log
I20260929 01:01:08.068341 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:08.059324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:58811 (local address 127.0.0.1:41793)
0929 01:01:08.059780 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:08.059802 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:08.059853 (+ 51us) server_negotiation.cc:407] Connection header received
0929 01:01:08.060700 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:08.060724 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:08.061173 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:08.061567 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:08.062016 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.063382 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.064343 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.065055 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.065211 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:08.066109 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:08.066115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:08.066132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:08.066678 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:08.066759 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:08.066791 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:08.066928 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:08.067117 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 01:01:08.067267 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":122,"threads_started":1}
I20260929 01:01:08.069017 21802 raft_consensus.cc:348] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02cab604c4746aeae579398a42e5010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41793 } }
I20260929 01:01:08.069332 21802 raft_consensus.cc:374] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:08.069394 21802 raft_consensus.cc:729] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02cab604c4746aeae579398a42e5010, State: Initialized, Role: FOLLOWER
I20260929 01:01:08.069854 21802 consensus_queue.cc:260] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02cab604c4746aeae579398a42e5010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41793 } }
I20260929 01:01:08.070008 21802 raft_consensus.cc:388] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:08.070075 21802 raft_consensus.cc:482] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:08.070225 21802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:08.071420 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:08.071385 21802 raft_consensus.cc:504] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02cab604c4746aeae579398a42e5010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41793 } }
I20260929 01:01:08.071729 21802 leader_election.cc:302] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e02cab604c4746aeae579398a42e5010; no voters:
I20260929 01:01:08.072070 21802 leader_election.cc:288] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:08.072180 21807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:08.072443 21807 raft_consensus.cc:686] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [term 1 LEADER]: Becoming Leader. State: Replica: e02cab604c4746aeae579398a42e5010, State: Running, Role: LEADER
I20260929 01:01:08.072739 21807 consensus_queue.cc:237] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02cab604c4746aeae579398a42e5010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41793 } }
I20260929 01:01:08.073415 21802 sys_catalog.cc:564] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:08.073868 21807 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e02cab604c4746aeae579398a42e5010. Latest consensus state: current_term: 1 leader_uuid: "e02cab604c4746aeae579398a42e5010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02cab604c4746aeae579398a42e5010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41793 } } }
I20260929 01:01:08.073980 21807 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:08.074425 21808 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e02cab604c4746aeae579398a42e5010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02cab604c4746aeae579398a42e5010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41793 } } }
I20260929 01:01:08.074519 21808 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010 [sys.catalog]: This master's current role is: LEADER
W20260929 01:01:08.078714 21827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:08.078817 21827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:08.078917 21815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:08.079785 21815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:08.081041 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:08.076118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:58395 (local address 127.0.0.1:41793)
0929 01:01:08.076247 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:08.076252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:08.076608 (+ 356us) server_negotiation.cc:407] Connection header received
0929 01:01:08.076700 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:08.076703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:08.076754 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:08.076835 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:08.077167 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.077921 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.079113 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.079447 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.079558 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:08.080537 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:08.080541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:08.080544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:08.080557 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:08.080585 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:08.080600 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:08.080691 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:08.080885 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 01:01:08.080919 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:01:08.085469 21815 catalog_manager.cc:1347] Generated new cluster ID: ec124a1a324d47d7a8b3f9c58e2d950f
I20260929 01:01:08.085552 21815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:08.099102 21815 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:08.100049 21815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:08.106341 21815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e02cab604c4746aeae579398a42e5010: Generated new TSK 0
I20260929 01:01:08.107115 21815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:08.119163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21734
W20260929 01:01:08.124552 21719 connection.cc:535] client connection to 127.0.0.1:41793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41793): Connection reset by peer (error 104)
2026-09-29T01:01:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:08 dist-test-slave-30s6 krb5kdc[21833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:08 dist-test-slave-30s6 krb5kdc[21833](info): set up 2 sockets
Sep 29 01:01:08 dist-test-slave-30s6 krb5kdc[21833](info): commencing operation
krb5kdc: starting...
W20260929 01:01:09.163007 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:09 dist-test-slave-30s6 krb5kdc[21833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:01:09.281885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:09.389544 21846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:09.389850 21846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:09.389931 21846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:09.393975 21846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:09.394065 21846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:09.394111 21846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:09.394164 21846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:09.394222 21846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:09.399477 21846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:09.400497 21846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:09.401501 21846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:09.410082 21851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:09.410084 21854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:09.410101 21846 server_base.cc:1034] running on GCE node
W20260929 01:01:09.410122 21852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:09.410948 21846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:09.412035 21846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:09.413229 21846 hybrid_clock.cc:648] HybridClock initialized: now 1790643669413185 us; error 68 us; skew 500 ppm
Sep 29 01:01:09 dist-test-slave-30s6 krb5kdc[21833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:09.415958 21846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:01:09.416810 21846 webserver.cc:460] Webserver started at http://127.0.0.1:36363/ using document root <none> and password file <none>
I20260929 01:01:09.417348 21846 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:09.417435 21846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:09.417659 21846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:09.419674 21846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "5babfdf9a1e241119375538744683fe8"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-30s6"
server_key: "7773abc98c44718ef0d80860f495176a"
server_key_iv: "d3edf87a59040c301b992aad76189c61"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.420230 21846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "5babfdf9a1e241119375538744683fe8"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-30s6"
server_key: "7773abc98c44718ef0d80860f495176a"
server_key_iv: "d3edf87a59040c301b992aad76189c61"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.423959 21846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:01:09.426366 21861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:09.427559 21846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:01:09.427656 21846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "5babfdf9a1e241119375538744683fe8"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-30s6"
server_key: "7773abc98c44718ef0d80860f495176a"
server_key_iv: "d3edf87a59040c301b992aad76189c61"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.427743 21846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:09.432516 21846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:09.435798 21846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:09.435956 21846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:09.444772 21846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34941
I20260929 01:01:09.444803 21913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34941 every 8 connection(s)
I20260929 01:01:09.445734 21846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:09.447544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21846
I20260929 01:01:09.447701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:09.447973 392 external_mini_cluster.cc:1428] Setting key 5d5981e3a66e5ba4daf2224adebf3d40
I20260929 01:01:09.448709 21914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:09.453462 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:09.449260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:38927 (local address 127.0.0.1:34941)
0929 01:01:09.449718 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:09.449736 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:09.449777 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:01:09.450636 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:09.450659 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:09.451023 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:09.451391 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:09.452085 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:09.452090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:09.452105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:09.452116 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:09.452177 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:09.452205 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:09.452348 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:09.452430 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 01:01:09.452579 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260929 01:01:09.455567 21914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8: Bootstrap starting.
I20260929 01:01:09.456173 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:09.457769 21914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:09.458612 21914 log.cc:824] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:09.459892 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:09.458235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:39631 (local address 127.0.0.1:34941)
0929 01:01:09.458370 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:09.458374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:09.458476 (+ 102us) server_negotiation.cc:407] Connection header received
0929 01:01:09.458618 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:09.458621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:09.458651 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:09.458735 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:09.459506 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:09.459510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:09.459512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:09.459524 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:09.459550 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:09.459567 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:09.459647 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:09.459739 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 01:01:09.459778 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:01:09.460572 21914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8: No bootstrap required, opened a new log
I20260929 01:01:09.464308 21914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5babfdf9a1e241119375538744683fe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260929 01:01:09.464516 21914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:09.464587 21914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5babfdf9a1e241119375538744683fe8, State: Initialized, Role: FOLLOWER
I20260929 01:01:09.464984 21914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5babfdf9a1e241119375538744683fe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260929 01:01:09.465116 21914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:09.465175 21914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:09.465261 21914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:09.466059 21914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5babfdf9a1e241119375538744683fe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260929 01:01:09.466408 21914 leader_election.cc:302] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5babfdf9a1e241119375538744683fe8; no voters:
I20260929 01:01:09.466681 21914 leader_election.cc:288] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:09.466862 21924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:09.467098 21924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [term 1 LEADER]: Becoming Leader. State: Replica: 5babfdf9a1e241119375538744683fe8, State: Running, Role: LEADER
I20260929 01:01:09.467451 21924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5babfdf9a1e241119375538744683fe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260929 01:01:09.467878 21914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:09.469084 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5babfdf9a1e241119375538744683fe8. Latest consensus state: current_term: 1 leader_uuid: "5babfdf9a1e241119375538744683fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5babfdf9a1e241119375538744683fe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } } }
I20260929 01:01:09.469220 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:09.469631 21933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:09.469661 21925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5babfdf9a1e241119375538744683fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5babfdf9a1e241119375538744683fe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } } }
I20260929 01:01:09.470218 21925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:09.472781 21933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:09.477835 21933 catalog_manager.cc:1347] Generated new cluster ID: 0f0201be7ea04814965508ed8acf5362
I20260929 01:01:09.477898 21933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:09.486701 21933 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:09.486992 21933 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:09.496138 21933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5babfdf9a1e241119375538744683fe8: Generated new TSK 0
I20260929 01:01:09.496796 21933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:09.504510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21846
2026-09-29T01:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:09 dist-test-slave-30s6 krb5kdc[21945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:09 dist-test-slave-30s6 krb5kdc[21945](info): set up 2 sockets
Sep 29 01:01:09 dist-test-slave-30s6 krb5kdc[21945](info): commencing operation
krb5kdc: starting...
W20260929 01:01:10.534462 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:10 dist-test-slave-30s6 krb5kdc[21945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:01:10.657269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:10.763947 21958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:10.764189 21958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:10.764228 21958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:10.768121 21958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:10.768170 21958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:10.768189 21958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:10.768206 21958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:10.768220 21958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:10.773330 21958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:10.774302 21958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:10.775269 21958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:10.783964 21964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:10.784050 21966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:10.783964 21963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:10.784278 21958 server_base.cc:1034] running on GCE node
I20260929 01:01:10.784974 21958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:10.786448 21958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:10.787685 21958 hybrid_clock.cc:648] HybridClock initialized: now 1790643670787665 us; error 36 us; skew 500 ppm
Sep 29 01:01:10 dist-test-slave-30s6 krb5kdc[21945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:10.790671 21958 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:01:10.791569 21958 webserver.cc:460] Webserver started at http://127.0.98.62:41517/ using document root <none> and password file <none>
I20260929 01:01:10.792115 21958 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:10.792197 21958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:10.792402 21958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:10.794485 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "acf9bf70c0d74e33a92d07c08032441f"
format_stamp: "Formatted at 2026-09-29 01:01:10 on dist-test-slave-30s6"
server_key: "e9e22ee26e971d0b0f6e53167f6dea42"
server_key_iv: "9c68a9540a7e6793cdcaaf80d5991ae5"
server_key_version: "encryptionkey@0"
I20260929 01:01:10.795032 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "acf9bf70c0d74e33a92d07c08032441f"
format_stamp: "Formatted at 2026-09-29 01:01:10 on dist-test-slave-30s6"
server_key: "e9e22ee26e971d0b0f6e53167f6dea42"
server_key_iv: "9c68a9540a7e6793cdcaaf80d5991ae5"
server_key_version: "encryptionkey@0"
I20260929 01:01:10.798715 21958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:01:10.801112 21973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:10.802464 21958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:01:10.802591 21958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "acf9bf70c0d74e33a92d07c08032441f"
format_stamp: "Formatted at 2026-09-29 01:01:10 on dist-test-slave-30s6"
server_key: "e9e22ee26e971d0b0f6e53167f6dea42"
server_key_iv: "9c68a9540a7e6793cdcaaf80d5991ae5"
server_key_version: "encryptionkey@0"
I20260929 01:01:10.802701 21958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:10.815084 21958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:10.818581 21958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:10.818786 21958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:10.825989 21958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37977
I20260929 01:01:10.826009 22025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37977 every 8 connection(s)
I20260929 01:01:10.827008 21958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:10.829862 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:10.832872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 21958
I20260929 01:01:10.833025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:10.833293 392 external_mini_cluster.cc:1428] Setting key c3c804c844bd37212544793c5547c068
I20260929 01:01:10.836241 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f: Bootstrap starting.
I20260929 01:01:10.838459 22026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:10.839180 22026 log.cc:824] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:10.841179 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f: No bootstrap required, opened a new log
Sep 29 01:01:10 dist-test-slave-30s6 krb5kdc[21945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:10.844964 22026 raft_consensus.cc:348] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf9bf70c0d74e33a92d07c08032441f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37977 } }
I20260929 01:01:10.845180 22026 raft_consensus.cc:374] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:10.845239 22026 raft_consensus.cc:729] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf9bf70c0d74e33a92d07c08032441f, State: Initialized, Role: FOLLOWER
I20260929 01:01:10.845670 22026 consensus_queue.cc:260] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf9bf70c0d74e33a92d07c08032441f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37977 } }
I20260929 01:01:10.845810 22026 raft_consensus.cc:388] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:10.845882 22026 raft_consensus.cc:482] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:10.845985 22026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:10.847064 22026 raft_consensus.cc:504] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf9bf70c0d74e33a92d07c08032441f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37977 } }
I20260929 01:01:10.847441 22026 leader_election.cc:302] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acf9bf70c0d74e33a92d07c08032441f; no voters:
I20260929 01:01:10.847793 22026 leader_election.cc:288] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:10.847882 22031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:10.848183 22031 raft_consensus.cc:686] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [term 1 LEADER]: Becoming Leader. State: Replica: acf9bf70c0d74e33a92d07c08032441f, State: Running, Role: LEADER
I20260929 01:01:10.848538 22031 consensus_queue.cc:237] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf9bf70c0d74e33a92d07c08032441f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37977 } }
I20260929 01:01:10.849013 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.834791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59009 (local address 127.0.98.62:37977)
0929 01:01:10.835288 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.835304 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.835342 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:01:10.836021 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.836039 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.836446 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.836788 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.837583 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.838672 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.839589 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.840181 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.840339 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.843181 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.843186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.843200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.843226 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.845695 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.846349 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.846354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.846358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.846487 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.846817 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.846820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.846821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.847245 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.847435 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.847776 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.848021 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":150,"threads_started":1}
I20260929 01:01:10.849184 22026 sys_catalog.cc:564] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:10.851791 22033 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [sys.catalog]: SysCatalogTable state changed. Reason: New leader acf9bf70c0d74e33a92d07c08032441f. Latest consensus state: current_term: 1 leader_uuid: "acf9bf70c0d74e33a92d07c08032441f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf9bf70c0d74e33a92d07c08032441f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37977 } } }
I20260929 01:01:10.851912 22032 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acf9bf70c0d74e33a92d07c08032441f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf9bf70c0d74e33a92d07c08032441f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37977 } } }
I20260929 01:01:10.852249 22033 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:10.852249 22032 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:10.852532 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 01:01:10.854866 22046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:10.854943 22046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:10.855048 22047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:10.856024 22047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:10.862131 22047 catalog_manager.cc:1347] Generated new cluster ID: c411c1123d634708b63a61fa0701b472
I20260929 01:01:10.862216 22047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:10.863238 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.854442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59239 (local address 127.0.98.62:37977)
0929 01:01:10.854569 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.854574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.854795 (+ 221us) server_negotiation.cc:407] Connection header received
0929 01:01:10.854927 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.854930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.854978 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.855049 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.855889 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.856468 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.857308 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.857542 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.857645 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.859365 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.859371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.859375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.859407 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.861251 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.861847 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.861850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.861852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.861939 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.862363 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.862366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.862368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.862546 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.862697 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.862971 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.863067 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 01:01:10.876869 22047 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:10.877802 22047 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:10.887051 22047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acf9bf70c0d74e33a92d07c08032441f: Generated new TSK 0
I20260929 01:01:10.887975 22047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:10.900453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 21958
W20260929 01:01:10.905208 21943 connection.cc:535] client connection to 127.0.98.62:37977 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37977): Connection reset by peer (error 104)
2026-09-29T01:01:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:10 dist-test-slave-30s6 krb5kdc[22057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:10 dist-test-slave-30s6 krb5kdc[22057](info): set up 2 sockets
Sep 29 01:01:10 dist-test-slave-30s6 krb5kdc[22057](info): commencing operation
krb5kdc: starting...
W20260929 01:01:12.010511 392 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:12 dist-test-slave-30s6 krb5kdc[22057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:01:12.129379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33207
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:12.236860 22070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:12.237112 22070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:12.237155 22070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:12.241472 22070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:12.241542 22070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:12.241564 22070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:12.241580 22070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:12.241595 22070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:12.246834 22070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:12.247769 22070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:12.248692 22070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:12.257140 22078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:12.257112 22076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:12.257201 22070 server_base.cc:1034] running on GCE node
W20260929 01:01:12.257117 22075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:12.257961 22070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:12.259028 22070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:12.260191 22070 hybrid_clock.cc:648] HybridClock initialized: now 1790643672260172 us; error 30 us; skew 500 ppm
Sep 29 01:01:12 dist-test-slave-30s6 krb5kdc[22057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:12.262904 22070 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:01:12.263720 22070 webserver.cc:460] Webserver started at http://127.0.98.62:40585/ using document root <none> and password file <none>
I20260929 01:01:12.264245 22070 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:12.264304 22070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:12.264540 22070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:12.266535 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "2a850d6b43234f83b5347e0144d9c177"
format_stamp: "Formatted at 2026-09-29 01:01:12 on dist-test-slave-30s6"
server_key: "4d2d1586adf589581524dae3361865b2"
server_key_iv: "192a4c394d3aa5f68f8db9e3907b2555"
server_key_version: "encryptionkey@0"
I20260929 01:01:12.267072 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a850d6b43234f83b5347e0144d9c177"
format_stamp: "Formatted at 2026-09-29 01:01:12 on dist-test-slave-30s6"
server_key: "4d2d1586adf589581524dae3361865b2"
server_key_iv: "192a4c394d3aa5f68f8db9e3907b2555"
server_key_version: "encryptionkey@0"
I20260929 01:01:12.270574 22070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:12.272843 22085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:12.273995 22070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:01:12.274134 22070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "2a850d6b43234f83b5347e0144d9c177"
format_stamp: "Formatted at 2026-09-29 01:01:12 on dist-test-slave-30s6"
server_key: "4d2d1586adf589581524dae3361865b2"
server_key_iv: "192a4c394d3aa5f68f8db9e3907b2555"
server_key_version: "encryptionkey@0"
I20260929 01:01:12.274281 22070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:12.291494 22070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:12.294730 22070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:12.294926 22070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:12.301765 22070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33207
I20260929 01:01:12.301792 22137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33207 every 8 connection(s)
I20260929 01:01:12.302724 22070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:12.305183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 22070
I20260929 01:01:12.305330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:12.305557 22138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:12.305624 392 external_mini_cluster.cc:1428] Setting key 67073fac87dfa3723f0ef0c91c324f98
I20260929 01:01:12.312384 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177: Bootstrap starting.
I20260929 01:01:12.314651 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:12.306934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59631 (local address 127.0.98.62:33207)
0929 01:01:12.307306 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:12.307324 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:12.307368 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:01:12.308217 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:12.308233 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:12.308544 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:12.308801 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:12.309140 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:12.310067 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:12.311293 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:12.311771 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:12.311912 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:12.312869 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:12.312874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:12.312886 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:12.313262 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:12.313315 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:12.313374 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:12.313483 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:12.313717 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 01:01:12.313860 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
I20260929 01:01:12.314775 22138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:12.315640 22138 log.cc:824] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:12.317705 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177: No bootstrap required, opened a new log
I20260929 01:01:12.317826 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:12.321763 22138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a850d6b43234f83b5347e0144d9c177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } }
I20260929 01:01:12.322034 22138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:12.322110 22138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a850d6b43234f83b5347e0144d9c177, State: Initialized, Role: FOLLOWER
I20260929 01:01:12.322620 22138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a850d6b43234f83b5347e0144d9c177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } }
I20260929 01:01:12.322795 22138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:12.322875 22138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:12.322970 22138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:12.323863 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:12.319874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36515 (local address 127.0.98.62:33207)
0929 01:01:12.319984 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:12.319990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:12.320111 (+ 121us) server_negotiation.cc:407] Connection header received
0929 01:01:12.320219 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:12.320221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:12.320255 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:12.320318 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:12.320583 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:12.321062 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:12.322162 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:12.322468 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:12.322553 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:12.323401 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:12.323404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:12.323406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:12.323416 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:12.323438 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:12.323448 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:12.323516 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:12.323664 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 01:01:12.323692 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:01:12.324084 22138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a850d6b43234f83b5347e0144d9c177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } }
I20260929 01:01:12.324412 22138 leader_election.cc:302] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a850d6b43234f83b5347e0144d9c177; no voters:
I20260929 01:01:12.324730 22138 leader_election.cc:288] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:12.324798 22148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:12.325002 22148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [term 1 LEADER]: Becoming Leader. State: Replica: 2a850d6b43234f83b5347e0144d9c177, State: Running, Role: LEADER
I20260929 01:01:12.325302 22148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a850d6b43234f83b5347e0144d9c177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } }
I20260929 01:01:12.325964 22138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:12.327013 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a850d6b43234f83b5347e0144d9c177. Latest consensus state: current_term: 1 leader_uuid: "2a850d6b43234f83b5347e0144d9c177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a850d6b43234f83b5347e0144d9c177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } } }
I20260929 01:01:12.327047 22149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a850d6b43234f83b5347e0144d9c177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a850d6b43234f83b5347e0144d9c177" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } } }
I20260929 01:01:12.327200 22149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:12.327464 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:12.327603 22157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:12.330560 22157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:12.336314 22157 catalog_manager.cc:1347] Generated new cluster ID: 3d4946ea134043cba1d0904ed10e4e4c
I20260929 01:01:12.336388 22157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:12.359217 22157 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:12.360381 22157 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:12.364997 22157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a850d6b43234f83b5347e0144d9c177: Generated new TSK 0
I20260929 01:01:12.365774 22157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:12.372336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 22070
W20260929 01:01:12.377827 22055 connection.cc:535] client connection to 127.0.98.62:33207 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33207): Connection reset by peer (error 104)
2026-09-29T01:01:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:12 dist-test-slave-30s6 krb5kdc[22169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:12 dist-test-slave-30s6 krb5kdc[22169](info): set up 2 sockets
Sep 29 01:01:12 dist-test-slave-30s6 krb5kdc[22169](info): commencing operation
krb5kdc: starting...
W20260929 01:01:13.410495 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:13 dist-test-slave-30s6 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:01:13.531627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:13.643553 22182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:13.643841 22182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:13.643911 22182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:13.648021 22182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:13.648108 22182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:13.648147 22182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:13.648183 22182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:13.648217 22182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:13.653427 22182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:13.654479 22182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:13.655491 22182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:13.663851 22190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:13.663858 22187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:13.663882 22188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:13.664120 22182 server_base.cc:1034] running on GCE node
I20260929 01:01:13.664840 22182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:13.666009 22182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:13.667203 22182 hybrid_clock.cc:648] HybridClock initialized: now 1790643673667201 us; error 46 us; skew 500 ppm
Sep 29 01:01:13 dist-test-slave-30s6 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:13.681051 22182 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:01:13.682099 22182 webserver.cc:460] Webserver started at http://127.0.98.62:38691/ using document root <none> and password file <none>
I20260929 01:01:13.682726 22182 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:13.682796 22182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:13.682974 22182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:13.685129 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "fdf26372fd444badac0ff4794aa78334"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-30s6"
server_key: "f7d12f7294ecd14ac6a5945c4a2072d2"
server_key_iv: "4b54442b564b96005c700d9bd1200bfe"
server_key_version: "encryptionkey@0"
I20260929 01:01:13.685688 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdf26372fd444badac0ff4794aa78334"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-30s6"
server_key: "f7d12f7294ecd14ac6a5945c4a2072d2"
server_key_iv: "4b54442b564b96005c700d9bd1200bfe"
server_key_version: "encryptionkey@0"
I20260929 01:01:13.689663 22182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 01:01:13.692224 22197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:13.693491 22182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:01:13.693627 22182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "fdf26372fd444badac0ff4794aa78334"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-30s6"
server_key: "f7d12f7294ecd14ac6a5945c4a2072d2"
server_key_iv: "4b54442b564b96005c700d9bd1200bfe"
server_key_version: "encryptionkey@0"
I20260929 01:01:13.693733 22182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:13.698781 22182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:13.702131 22182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:13.702397 22182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:13.710421 22182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36963
I20260929 01:01:13.710458 22249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36963 every 8 connection(s)
I20260929 01:01:13.711476 22182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:13.714243 22250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:13.717656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 22182
I20260929 01:01:13.717797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:13.718101 392 external_mini_cluster.cc:1428] Setting key ddfb0558bec6fb60ec8fbe76600a58f8
I20260929 01:01:13.720084 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334: Bootstrap starting.
I20260929 01:01:13.722493 22250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:13.723289 22250 log.cc:824] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:13.723829 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:13.719596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41075 (local address 127.0.98.62:36963)
0929 01:01:13.720087 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:13.720103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:13.720145 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:01:13.720878 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:13.720896 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:13.721192 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:13.721510 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:13.722392 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:13.722397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:13.722410 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:13.722423 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:13.722480 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:13.722502 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:13.722642 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:13.722788 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 01:01:13.722982 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":153,"threads_started":1}
I20260929 01:01:13.725198 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334: No bootstrap required, opened a new log
I20260929 01:01:13.726730 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:13.729571 22250 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf26372fd444badac0ff4794aa78334" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36963 } }
I20260929 01:01:13.729842 22250 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:13.729914 22250 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf26372fd444badac0ff4794aa78334, State: Initialized, Role: FOLLOWER
I20260929 01:01:13.730404 22250 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf26372fd444badac0ff4794aa78334" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36963 } }
I20260929 01:01:13.730581 22250 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:13.730659 22250 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:13.730769 22250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:13.731006 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:13.729079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56251 (local address 127.0.98.62:36963)
0929 01:01:13.729217 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:13.729221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:13.729464 (+ 243us) server_negotiation.cc:407] Connection header received
0929 01:01:13.729561 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:13.729565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:13.729600 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:13.729688 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:13.730482 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:13.730485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:13.730488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:13.730498 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:13.730526 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:13.730541 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:13.730644 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:13.730828 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 01:01:13.730871 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:01:13.731714 22250 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf26372fd444badac0ff4794aa78334" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36963 } }
I20260929 01:01:13.732048 22250 leader_election.cc:302] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf26372fd444badac0ff4794aa78334; no voters:
I20260929 01:01:13.732333 22250 leader_election.cc:288] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:13.732461 22260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:13.732683 22260 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [term 1 LEADER]: Becoming Leader. State: Replica: fdf26372fd444badac0ff4794aa78334, State: Running, Role: LEADER
I20260929 01:01:13.733042 22260 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf26372fd444badac0ff4794aa78334" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36963 } }
I20260929 01:01:13.733620 22250 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:13.734855 22262 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf26372fd444badac0ff4794aa78334. Latest consensus state: current_term: 1 leader_uuid: "fdf26372fd444badac0ff4794aa78334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf26372fd444badac0ff4794aa78334" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36963 } } }
I20260929 01:01:13.734863 22261 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf26372fd444badac0ff4794aa78334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf26372fd444badac0ff4794aa78334" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36963 } } }
I20260929 01:01:13.734987 22261 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:13.734989 22262 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:13.735435 22269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:13.738288 22269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:13.743419 22269 catalog_manager.cc:1347] Generated new cluster ID: 48fffb9f4e8c46d2b439d0b5e9912e13
I20260929 01:01:13.743479 22269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:13.772902 22269 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:13.773223 22269 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:13.783979 22269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf26372fd444badac0ff4794aa78334: Generated new TSK 0
I20260929 01:01:13.784696 22269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:13.792536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 22182
2026-09-29T01:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:13 dist-test-slave-30s6 krb5kdc[22281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:13 dist-test-slave-30s6 krb5kdc[22281](info): set up 2 sockets
Sep 29 01:01:13 dist-test-slave-30s6 krb5kdc[22281](info): commencing operation
krb5kdc: starting...
W20260929 01:01:14.826566 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:14 dist-test-slave-30s6 krb5kdc[22281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:01:14.945468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:15.053424 22294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:15.053716 22294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:15.053795 22294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:15.057737 22294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:15.057821 22294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:15.057871 22294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:15.057924 22294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:15.057958 22294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:15.063094 22294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:15.064121 22294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:15.065125 22294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:15.073545 22300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:15.073634 22299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:15.073567 22294 server_base.cc:1034] running on GCE node
W20260929 01:01:15.073621 22302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:15.074368 22294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:15.075491 22294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:15.076684 22294 hybrid_clock.cc:648] HybridClock initialized: now 1790643675076653 us; error 47 us; skew 500 ppm
Sep 29 01:01:15 dist-test-slave-30s6 krb5kdc[22281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:15.079527 22294 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:01:15.080402 22294 webserver.cc:460] Webserver started at http://127.0.98.62:45661/ using document root <none> and password file <none>
I20260929 01:01:15.080979 22294 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:15.081067 22294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:15.081287 22294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:15.083439 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "ae834df30a5449ceb2ac492fe65672a5"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-30s6"
server_key: "525a591af84962091654494642d8c845"
server_key_iv: "71a85f984ef367b58de6428a57791ab1"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.084016 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae834df30a5449ceb2ac492fe65672a5"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-30s6"
server_key: "525a591af84962091654494642d8c845"
server_key_iv: "71a85f984ef367b58de6428a57791ab1"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.087716 22294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:01:15.090137 22309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:15.091423 22294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:01:15.091573 22294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "ae834df30a5449ceb2ac492fe65672a5"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-30s6"
server_key: "525a591af84962091654494642d8c845"
server_key_iv: "71a85f984ef367b58de6428a57791ab1"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.091691 22294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:15.102608 22294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:15.105635 22294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:15.105835 22294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:15.112914 22294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35069
I20260929 01:01:15.112939 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35069 every 8 connection(s)
I20260929 01:01:15.113909 22294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:15.116665 22362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:15.121578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 22294
I20260929 01:01:15.121726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:15.122008 392 external_mini_cluster.cc:1428] Setting key 78707330d26348233c7e636c68f2e26f
I20260929 01:01:15.122226 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5: Bootstrap starting.
I20260929 01:01:15.124730 22362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:15.125489 22362 log.cc:824] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:15.127748 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5: No bootstrap required, opened a new log
Sep 29 01:01:15 dist-test-slave-30s6 krb5kdc[22281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:15.132108 22362 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae834df30a5449ceb2ac492fe65672a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35069 } }
I20260929 01:01:15.132397 22362 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:15.132526 22362 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae834df30a5449ceb2ac492fe65672a5, State: Initialized, Role: FOLLOWER
I20260929 01:01:15.133008 22362 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae834df30a5449ceb2ac492fe65672a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35069 } }
I20260929 01:01:15.133147 22362 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:15.133216 22362 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:15.133338 22362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:15.134230 22362 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae834df30a5449ceb2ac492fe65672a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35069 } }
I20260929 01:01:15.134568 22362 leader_election.cc:302] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae834df30a5449ceb2ac492fe65672a5; no voters:
I20260929 01:01:15.134887 22362 leader_election.cc:288] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:15.134966 22367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:15.135201 22367 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [term 1 LEADER]: Becoming Leader. State: Replica: ae834df30a5449ceb2ac492fe65672a5, State: Running, Role: LEADER
I20260929 01:01:15.135565 22367 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae834df30a5449ceb2ac492fe65672a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35069 } }
I20260929 01:01:15.136212 22362 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:15.137593 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.123347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50769 (local address 127.0.98.62:35069)
0929 01:01:15.123826 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.123841 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.123877 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:01:15.124521 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.124538 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.124938 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.125310 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.126161 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.127627 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.128526 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.129213 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.129382 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.132134 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.132140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.132157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.132190 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.134044 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.134555 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.134561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.134570 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.134701 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.135018 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.135024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.135027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.135562 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.135731 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.136214 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.136534 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":137,"threads_started":1}
I20260929 01:01:15.138860 22369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae834df30a5449ceb2ac492fe65672a5. Latest consensus state: current_term: 1 leader_uuid: "ae834df30a5449ceb2ac492fe65672a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae834df30a5449ceb2ac492fe65672a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35069 } } }
I20260929 01:01:15.139285 22369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:15.139057 22368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae834df30a5449ceb2ac492fe65672a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae834df30a5449ceb2ac492fe65672a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35069 } } }
I20260929 01:01:15.139503 22368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:15.140630 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 01:01:15.141636 22382 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:15.141710 22382 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:15.141788 22383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:15.142558 22383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:15.147603 22383 catalog_manager.cc:1347] Generated new cluster ID: 58bbf2290f6a4306bb55e594b51a4ce5
I20260929 01:01:15.147660 22383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:15.153792 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.146229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37873 (local address 127.0.98.62:35069)
0929 01:01:15.146320 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.146324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.146586 (+ 262us) server_negotiation.cc:407] Connection header received
0929 01:01:15.146759 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.146762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.146806 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.146854 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.147798 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.148351 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.149131 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.149378 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.149487 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.150865 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.150869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.150871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.150894 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.152270 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.152671 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.152676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.152678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.152753 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.153018 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.153021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.153023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.153183 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.153309 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.153530 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.153639 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 01:01:15.165395 22383 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:15.166344 22383 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:15.179697 22383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae834df30a5449ceb2ac492fe65672a5: Generated new TSK 0
I20260929 01:01:15.180418 22383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:15.189411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 22294
2026-09-29T01:01:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:15 dist-test-slave-30s6 krb5kdc[22393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:15 dist-test-slave-30s6 krb5kdc[22393](info): set up 2 sockets
Sep 29 01:01:15 dist-test-slave-30s6 krb5kdc[22393](info): commencing operation
krb5kdc: starting...
W20260929 01:01:16.234902 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:16 dist-test-slave-30s6 krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:01:16.353510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:16.465752 22406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:16.466161 22406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:16.466262 22406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:16.470613 22406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:16.470695 22406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:16.470731 22406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:16.470767 22406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:16.470800 22406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:16.476220 22406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:16.477342 22406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:16.478472 22406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:16.487555 22414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:16.487601 22412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:16.487576 22411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:16.487885 22406 server_base.cc:1034] running on GCE node
I20260929 01:01:16.488457 22406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:16.489619 22406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:16.490788 22406 hybrid_clock.cc:648] HybridClock initialized: now 1790643676490748 us; error 55 us; skew 500 ppm
Sep 29 01:01:16 dist-test-slave-30s6 krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:16.493594 22406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:01:16.494532 22406 webserver.cc:460] Webserver started at http://127.0.98.62:39177/ using document root <none> and password file <none>
I20260929 01:01:16.495066 22406 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:16.495126 22406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:16.495285 22406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:16.497332 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "eb7ffed95b1d445693e1f33f8bd8f314"
format_stamp: "Formatted at 2026-09-29 01:01:16 on dist-test-slave-30s6"
server_key: "1b9b50efac8cfcc89e2001e49f837e4c"
server_key_iv: "3603d48f93c5267d43fb9d9dfd57aa0a"
server_key_version: "encryptionkey@0"
I20260929 01:01:16.497833 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb7ffed95b1d445693e1f33f8bd8f314"
format_stamp: "Formatted at 2026-09-29 01:01:16 on dist-test-slave-30s6"
server_key: "1b9b50efac8cfcc89e2001e49f837e4c"
server_key_iv: "3603d48f93c5267d43fb9d9dfd57aa0a"
server_key_version: "encryptionkey@0"
I20260929 01:01:16.501502 22406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:01:16.504057 22421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:16.505352 22406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:01:16.505496 22406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "eb7ffed95b1d445693e1f33f8bd8f314"
format_stamp: "Formatted at 2026-09-29 01:01:16 on dist-test-slave-30s6"
server_key: "1b9b50efac8cfcc89e2001e49f837e4c"
server_key_iv: "3603d48f93c5267d43fb9d9dfd57aa0a"
server_key_version: "encryptionkey@0"
I20260929 01:01:16.505612 22406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:16.522068 22406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:16.525277 22406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:16.525481 22406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:16.533522 22406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44405
I20260929 01:01:16.533543 22473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44405 every 8 connection(s)
I20260929 01:01:16.534649 22406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:16.537494 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:16.540005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 22406
I20260929 01:01:16.540159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:16.540437 392 external_mini_cluster.cc:1428] Setting key 31b17ac586a6d6e2b40a2bceb5a95466
I20260929 01:01:16.543908 22474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314: Bootstrap starting.
I20260929 01:01:16.546550 22474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:16.547425 22474 log.cc:824] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:16.549628 22474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314: No bootstrap required, opened a new log
I20260929 01:01:16.550668 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.541881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52269 (local address 127.0.98.62:44405)
0929 01:01:16.542339 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.542359 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.542402 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:01:16.543202 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.543226 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.543672 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.544052 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.544483 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.545942 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.546933 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.547642 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.547818 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.548697 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.548703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.548719 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:16.549199 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.549267 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:16.549299 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.549394 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.549501 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.549711 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
I20260929 01:01:16.553464 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:16.553942 22474 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7ffed95b1d445693e1f33f8bd8f314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } }
I20260929 01:01:16.554209 22474 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:16.554287 22474 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7ffed95b1d445693e1f33f8bd8f314, State: Initialized, Role: FOLLOWER
I20260929 01:01:16.554769 22474 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7ffed95b1d445693e1f33f8bd8f314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } }
I20260929 01:01:16.554957 22474 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:16.555029 22474 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:16.555130 22474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:16.556285 22474 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7ffed95b1d445693e1f33f8bd8f314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } }
I20260929 01:01:16.556632 22474 leader_election.cc:302] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7ffed95b1d445693e1f33f8bd8f314; no voters:
I20260929 01:01:16.556957 22474 leader_election.cc:288] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:16.557029 22484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:16.557219 22484 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [term 1 LEADER]: Becoming Leader. State: Replica: eb7ffed95b1d445693e1f33f8bd8f314, State: Running, Role: LEADER
I20260929 01:01:16.557521 22484 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7ffed95b1d445693e1f33f8bd8f314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } }
I20260929 01:01:16.558292 22474 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:16.558682 22485 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7ffed95b1d445693e1f33f8bd8f314. Latest consensus state: current_term: 1 leader_uuid: "eb7ffed95b1d445693e1f33f8bd8f314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7ffed95b1d445693e1f33f8bd8f314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } }
I20260929 01:01:16.558715 22486 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7ffed95b1d445693e1f33f8bd8f314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7ffed95b1d445693e1f33f8bd8f314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } }
I20260929 01:01:16.558909 22485 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:16.558913 22486 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:16.559900 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.555688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48999 (local address 127.0.98.62:44405)
0929 01:01:16.555809 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.555813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.555978 (+ 165us) server_negotiation.cc:407] Connection header received
0929 01:01:16.556104 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.556109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.556156 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.556238 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.556503 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.557139 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.558049 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.558357 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.558452 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.559488 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.559492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.559495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:16.559507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.559537 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:16.559548 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.559624 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.559720 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.559768 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 01:01:16.562897 22499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:16.562985 22499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:16.563054 22490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:16.563766 22490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:16.569435 22490 catalog_manager.cc:1347] Generated new cluster ID: 3926954a45924d77b0e0322bc5da0a42
I20260929 01:01:16.569514 22490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:16.580221 22490 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:16.581535 22490 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:16.590514 22490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7ffed95b1d445693e1f33f8bd8f314: Generated new TSK 0
I20260929 01:01:16.591374 22490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:16.597636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 22406
2026-09-29T01:01:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:16 dist-test-slave-30s6 krb5kdc[22505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:16 dist-test-slave-30s6 krb5kdc[22505](info): set up 2 sockets
Sep 29 01:01:16 dist-test-slave-30s6 krb5kdc[22505](info): commencing operation
krb5kdc: starting...
W20260929 01:01:17.654466 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:17 dist-test-slave-30s6 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:01:17.777156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46495
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:17.885293 22518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:17.885586 22518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:17.885658 22518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:17.890288 22518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:17.890372 22518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:17.890410 22518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:17.890446 22518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:17.890479 22518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:17.896912 22518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:17.898092 22518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:17.899215 22518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:17.908111 22526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:17.908097 22523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:17.908278 22518 server_base.cc:1034] running on GCE node
W20260929 01:01:17.908080 22524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:17.908952 22518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:17.910029 22518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:17.911188 22518 hybrid_clock.cc:648] HybridClock initialized: now 1790643677911178 us; error 29 us; skew 500 ppm
Sep 29 01:01:17 dist-test-slave-30s6 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:17.913831 22518 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:01:17.914863 22518 webserver.cc:460] Webserver started at http://127.0.98.62:41155/ using document root <none> and password file <none>
I20260929 01:01:17.915434 22518 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:17.915529 22518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:17.915772 22518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:17.918032 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "507fb70f8b894e67beda3cdcd8739f66"
format_stamp: "Formatted at 2026-09-29 01:01:17 on dist-test-slave-30s6"
server_key: "bf1b19fec2494dcabdea3eeaf2894702"
server_key_iv: "4bd3d178dddffc835aed79438ca22f28"
server_key_version: "encryptionkey@0"
I20260929 01:01:17.918694 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "507fb70f8b894e67beda3cdcd8739f66"
format_stamp: "Formatted at 2026-09-29 01:01:17 on dist-test-slave-30s6"
server_key: "bf1b19fec2494dcabdea3eeaf2894702"
server_key_iv: "4bd3d178dddffc835aed79438ca22f28"
server_key_version: "encryptionkey@0"
I20260929 01:01:17.922580 22518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:01:17.925357 22533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:17.926708 22518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:17.926852 22518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "507fb70f8b894e67beda3cdcd8739f66"
format_stamp: "Formatted at 2026-09-29 01:01:17 on dist-test-slave-30s6"
server_key: "bf1b19fec2494dcabdea3eeaf2894702"
server_key_iv: "4bd3d178dddffc835aed79438ca22f28"
server_key_version: "encryptionkey@0"
I20260929 01:01:17.926978 22518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:17.931916 22518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:17.934860 22518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:17.935072 22518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:17.942484 22518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46495
I20260929 01:01:17.942512 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46495 every 8 connection(s)
I20260929 01:01:17.943470 22518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:17.946135 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:17.951332 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66: Bootstrap starting.
I20260929 01:01:17.952926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 22518
I20260929 01:01:17.953079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:17.953361 392 external_mini_cluster.cc:1428] Setting key 953133d4e86367e097c014c0d8a36d28
I20260929 01:01:17.953809 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:17.954757 22586 log.cc:824] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:17.956746 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66: No bootstrap required, opened a new log
I20260929 01:01:17.958634 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.954511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46313 (local address 127.0.98.62:46495)
0929 01:01:17.954952 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:17.954968 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:17.955004 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:01:17.955811 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:17.955829 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:17.956114 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:17.956407 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:17.957293 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:17.957298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:17.957309 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:17.957321 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:17.957385 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:17.957406 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:17.957504 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:17.957646 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 01:01:17.957813 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
I20260929 01:01:17.960891 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507fb70f8b894e67beda3cdcd8739f66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46495 } }
I20260929 01:01:17.961167 22586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:17.961242 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507fb70f8b894e67beda3cdcd8739f66, State: Initialized, Role: FOLLOWER
I20260929 01:01:17.961448 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:17.961675 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507fb70f8b894e67beda3cdcd8739f66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46495 } }
I20260929 01:01:17.961820 22586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:17.961898 22586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:17.962003 22586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:17.963074 22586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507fb70f8b894e67beda3cdcd8739f66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46495 } }
I20260929 01:01:17.963418 22586 leader_election.cc:302] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 507fb70f8b894e67beda3cdcd8739f66; no voters:
I20260929 01:01:17.963805 22586 leader_election.cc:288] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:17.963884 22596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:17.964164 22596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [term 1 LEADER]: Becoming Leader. State: Replica: 507fb70f8b894e67beda3cdcd8739f66, State: Running, Role: LEADER
I20260929 01:01:17.964519 22596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507fb70f8b894e67beda3cdcd8739f66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46495 } }
I20260929 01:01:17.965229 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:17.965601 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.963607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58721 (local address 127.0.98.62:46495)
0929 01:01:17.963760 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:17.963765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:17.963960 (+ 195us) server_negotiation.cc:407] Connection header received
0929 01:01:17.964106 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:17.964109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:17.964143 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:17.964220 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:17.965038 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:17.965042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:17.965045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:17.965056 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:17.965083 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:17.965095 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:17.965167 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:17.965428 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 01:01:17.965463 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:01:17.966223 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "507fb70f8b894e67beda3cdcd8739f66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507fb70f8b894e67beda3cdcd8739f66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46495 } } }
I20260929 01:01:17.966668 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:17.966944 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 507fb70f8b894e67beda3cdcd8739f66. Latest consensus state: current_term: 1 leader_uuid: "507fb70f8b894e67beda3cdcd8739f66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507fb70f8b894e67beda3cdcd8739f66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46495 } } }
I20260929 01:01:17.967075 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66 [sys.catalog]: This master's current role is: LEADER
W20260929 01:01:17.970302 22611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:17.970381 22611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:17.970484 22607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:17.971176 22607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:17.976454 22607 catalog_manager.cc:1347] Generated new cluster ID: 616fe86cc3774d77b00d62f04c01bf21
I20260929 01:01:17.976552 22607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:18.001227 22607 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:18.001595 22607 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:18.012758 22607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 507fb70f8b894e67beda3cdcd8739f66: Generated new TSK 0
I20260929 01:01:18.013515 22607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:18.027661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 22518
2026-09-29T01:01:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1430 ms)
[----------] 18 tests from AuthTokenIssuingTest (25436 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:18 dist-test-slave-30s6 krb5kdc[22617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:18 dist-test-slave-30s6 krb5kdc[22617](info): set up 2 sockets
Sep 29 01:01:18 dist-test-slave-30s6 krb5kdc[22617](info): commencing operation
krb5kdc: starting...
W20260929 01:01:19.066442 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:01:19.186898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36879,127.0.98.61:39379,127.0.98.60:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:19.302320 22630 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:01:19.302593 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:19.302618 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:19.302716 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:19.307080 22630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:19.307143 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:19.307163 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:19.307180 22630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:19.307235 22630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:19.312716 22630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36879,127.0.98.61:39379,127.0.98.60:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:19.313752 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:19.314810 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:01:19.323348 22630 server_base.cc:1034] running on GCE node
W20260929 01:01:19.323330 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:19.323523 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:19.323382 22638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:19.324102 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:19.325160 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:19.326352 22630 hybrid_clock.cc:648] HybridClock initialized: now 1790643679326336 us; error 39 us; skew 500 ppm
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:19.332615 22630 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:01:19.333539 22630 webserver.cc:460] Webserver started at http://127.0.98.62:40555/ using document root <none> and password file <none>
I20260929 01:01:19.334062 22630 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:19.334120 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:19.334344 22630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:19.336385 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "49ecb16605c542c5bbc0066d4f00a4df"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "0c9fcaecf28e7258b822ab1cab52f746"
server_key_iv: "1926331ac2f21ab530fb886e399218f6"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.336897 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "49ecb16605c542c5bbc0066d4f00a4df"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "0c9fcaecf28e7258b822ab1cab52f746"
server_key_iv: "1926331ac2f21ab530fb886e399218f6"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.340457 22630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:01:19.342859 22645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:19.344177 22630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:01:19.344281 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "49ecb16605c542c5bbc0066d4f00a4df"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "0c9fcaecf28e7258b822ab1cab52f746"
server_key_iv: "1926331ac2f21ab530fb886e399218f6"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.344363 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:19.357514 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:19.360610 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:19.360769 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:19.367744 22630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36879
I20260929 01:01:19.368017 22697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36879 every 8 connection(s)
I20260929 01:01:19.368681 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:19.371407 22698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:19.373008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 22630
I20260929 01:01:19.373168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:19.373432 392 external_mini_cluster.cc:1428] Setting key 26b5e0c6d8a45872920881368178dd6c
I20260929 01:01:19.375334 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:19.391669 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.377883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.0.98.62:36879)
0929 01:01:19.378977 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.378989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.379033 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:01:19.379742 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.379746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.379929 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.380118 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.380923 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.381851 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.383182 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.383779 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.383937 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.386786 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.386792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.386813 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.386833 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.388533 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.389037 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.389046 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.389052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.389150 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.389461 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.389464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.389466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.389830 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.389972 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.390347 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.390614 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":978,"spinlock_wait_cycles":23936,"thread_start_us":160,"threads_started":1}
I20260929 01:01:19.391669 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.377877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36879 (local address 127.0.0.1:36594)
0929 01:01:19.378385 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.378435 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.379425 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.380110 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.380115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.380528 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.380776 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.380789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.381973 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.381977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.383021 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.383046 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.383963 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.383970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.384131 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.384763 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.384790 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.386619 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.388663 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.388672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.388684 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.388922 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.389249 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.389252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.389254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.389367 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.389946 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.389953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.390093 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.390277 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.390663 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":108,"thread_start_us":133,"threads_started":1}
I20260929 01:01:19.394361 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:19.395139 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.394838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35160)
0929 01:01:19.394978 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.395050 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 01:01:19.395370 22649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
I20260929 01:01:19.396354 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.396019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35174)
0929 01:01:19.396239 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.396299 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 01:01:19.396516 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39379: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 01:01:19.401846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39379
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36879,127.0.98.61:39379,127.0.98.60:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:19.445796 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } attempt: 1
I20260929 01:01:19.446767 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.446433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35190)
0929 01:01:19.446604 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.446676 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 01:01:19.447503 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.447253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35194)
0929 01:01:19.447363 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.447437 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:01:19.447661 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39379: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
W20260929 01:01:19.512048 22705 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:01:19.512332 22705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:19.512355 22705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:19.512392 22705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 01:01:19.515843 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } attempt: 2
W20260929 01:01:19.516593 22705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:19.516649 22705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:19.516669 22705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:19.516706 22705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:19.516760 22705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:19.516742 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.516367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35196)
0929 01:01:19.516536 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.516604 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 01:01:19.517536 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.517315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35210)
0929 01:01:19.517439 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.517484 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 01:01:19.517679 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39379: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39379: connect: Connection refused (error 111)
I20260929 01:01:19.522614 22705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36879,127.0.98.61:39379,127.0.98.60:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:19.523694 22705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:19.524724 22705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:19.533429 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:19.533372 22710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:19.533406 22705 server_base.cc:1034] running on GCE node
W20260929 01:01:19.533372 22711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:19.534224 22705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:19.535342 22705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:19.536528 22705 hybrid_clock.cc:648] HybridClock initialized: now 1790643679536514 us; error 28 us; skew 500 ppm
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:19.539295 22705 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 01:01:19.540138 22705 webserver.cc:460] Webserver started at http://127.0.98.61:33437/ using document root <none> and password file <none>
I20260929 01:01:19.540632 22705 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:19.540686 22705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:19.540840 22705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:19.542793 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/data/instance:
uuid: "fa53b1e0a7824e09932ce413c1dff728"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "e743e617fe24f097b79bbf1c27465702"
server_key_iv: "e4313eaf8b66c170c1542a9e291d4b11"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.543279 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/wal/instance:
uuid: "fa53b1e0a7824e09932ce413c1dff728"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "e743e617fe24f097b79bbf1c27465702"
server_key_iv: "e4313eaf8b66c170c1542a9e291d4b11"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.546833 22705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:01:19.549117 22720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:19.550312 22705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:01:19.550405 22705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/wal
uuid: "fa53b1e0a7824e09932ce413c1dff728"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "e743e617fe24f097b79bbf1c27465702"
server_key_iv: "e4313eaf8b66c170c1542a9e291d4b11"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.550482 22705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:19.563776 22705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:19.566748 22705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:19.566895 22705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:19.573764 22705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39379
I20260929 01:01:19.574054 22772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39379 every 8 connection(s)
I20260929 01:01:19.574756 22705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
I20260929 01:01:19.577406 22773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:19.577881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 22705
I20260929 01:01:19.578013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-1/wal/instance
I20260929 01:01:19.578330 392 external_mini_cluster.cc:1428] Setting key cd69cc3dd40edabd9db195360d6c7d28
I20260929 01:01:19.582427 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:19.597707 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.584402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.0.98.62:36879)
0929 01:01:19.584545 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.584549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.585560 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 01:01:19.586607 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.586613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.586684 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.586787 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.587971 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.588589 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.590091 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.590390 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.590477 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.593580 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.593584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.593586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.593615 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.595477 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.595999 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.596003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.596005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.596125 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.596461 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.596465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.596467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.596722 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.596830 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.597383 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.597526 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:01:19.598568 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.584878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36879 (local address 127.0.0.1:36602)
0929 01:01:19.585383 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.585424 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.586347 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.586930 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.586936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.587280 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.587838 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.587852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.588702 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.588705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.589946 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.589961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.590544 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.590551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.590790 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.591435 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.591450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.593402 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.595608 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.595613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.595626 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.595890 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.596226 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.596229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.596230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.596366 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.596831 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.596836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.597113 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.597277 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.597547 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":141,"threads_started":1}
I20260929 01:01:19.600175 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 01:01:19.605358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39473
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36879,127.0.98.61:39379,127.0.98.60:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:01:19.608857 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } attempt: 3
I20260929 01:01:19.610116 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.600673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35218)
0929 01:01:19.600791 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.600813 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.600895 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.601465 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.601468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.601648 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.601719 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.601724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.602721 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.602727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.603269 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.603277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.603885 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.603888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.604009 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.604502 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.604523 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.605959 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.608336 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.608338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.608340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.608523 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.608866 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.608869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.608887 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.608981 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.609651 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.609654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.609768 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.609870 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.609995 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":5}
I20260929 01:01:19.610704 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.600796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.0.98.61:39379)
0929 01:01:19.601098 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.601106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.601123 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:01:19.601155 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.601158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.601303 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.601481 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.601824 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.602618 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.603413 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.603765 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.603879 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.606118 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.606126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.606139 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.606160 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.608211 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.608631 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.608640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.608645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.608740 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.609091 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.609098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.609102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.609533 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.609685 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.610384 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.610529 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":228,"thread_start_us":122,"threads_started":1}
I20260929 01:01:19.612757 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:19.613514 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.613224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50754)
0929 01:01:19.613367 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.613449 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 01:01:19.614370 22724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:01:19.615424 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.615128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50756)
0929 01:01:19.615290 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.615348 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260929 01:01:19.615614 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39473: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
I20260929 01:01:19.618940 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.609321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35222)
0929 01:01:19.609525 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.609540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.609645 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.610389 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.610393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.610567 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.610689 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.610698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.611896 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.611899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.613177 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.613187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.613770 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.613774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.613881 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.614415 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.614426 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.615725 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.617457 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.617462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.617464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.617656 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.617942 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.617947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.617949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.618033 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.618436 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.618439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.618542 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.618647 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.618780 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260929 01:01:19.618963 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.609343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.0.98.61:39379)
0929 01:01:19.610001 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.610012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.610028 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:19.610079 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.610084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.610139 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.610273 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.610823 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.611754 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.613296 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.613642 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.613768 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.615866 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.615870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.615872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.615909 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.617344 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.617754 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.617759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.617761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.617845 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.618122 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.618127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.618128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.618341 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.618446 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.618697 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.618800 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":412,"threads_started":1}
I20260929 01:01:19.619563 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:19.620223 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.620001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50770)
0929 01:01:19.620105 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.620155 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 01:01:19.620839 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.620617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50776)
0929 01:01:19.620746 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.620788 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 01:01:19.620951 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39473: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
I20260929 01:01:19.652098 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } attempt: 1
I20260929 01:01:19.652951 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.652640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50782)
0929 01:01:19.652806 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.652868 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 01:01:19.653823 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.653611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50796)
0929 01:01:19.653717 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.653769 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 01:01:19.653955 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39473: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
I20260929 01:01:19.664882 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } attempt: 1
I20260929 01:01:19.665654 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.665344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50808)
0929 01:01:19.665497 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.665569 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 01:01:19.666430 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.666172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50812)
0929 01:01:19.666312 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.666364 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 01:01:19.666611 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39473: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
W20260929 01:01:19.717792 22779 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:01:19.718084 22779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:19.718107 22779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:19.718221 22779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:19.722241 22779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:19.722294 22779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:19.722313 22779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:19.722329 22779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:19.722389 22779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:19.727638 22779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36879,127.0.98.61:39379,127.0.98.60:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:19.728715 22779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:19.729130 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } attempt: 2
I20260929 01:01:19.729965 22779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:01:19.729976 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.729685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50814)
0929 01:01:19.729835 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.729896 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 01:01:19.730751 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.730536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50826)
0929 01:01:19.730640 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.730696 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 01:01:19.730875 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39473: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
I20260929 01:01:19.734779 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } attempt: 2
I20260929 01:01:19.735527 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.735250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50838)
0929 01:01:19.735403 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.735456 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 01:01:19.736188 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.735968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50852)
0929 01:01:19.736073 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.736135 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:01:19.736312 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39473: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39473: connect: Connection refused (error 111)
W20260929 01:01:19.738943 22789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:19.738976 22786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:19.738947 22787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:19.739174 22779 server_base.cc:1034] running on GCE node
I20260929 01:01:19.739852 22779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:19.741008 22779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:19.742228 22779 hybrid_clock.cc:648] HybridClock initialized: now 1790643679742194 us; error 49 us; skew 500 ppm
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:19.744879 22779 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 01:01:19.745738 22779 webserver.cc:460] Webserver started at http://127.0.98.60:45405/ using document root <none> and password file <none>
I20260929 01:01:19.746337 22779 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:19.746471 22779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:19.746737 22779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:19.748929 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/data/instance:
uuid: "0ed36f5fe31e4b8b8bae557261a709cd"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "af93018384416d411f6e12a70ba6282c"
server_key_iv: "e7b2bb61a63895fe2746410b08e19c68"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.749478 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/wal/instance:
uuid: "0ed36f5fe31e4b8b8bae557261a709cd"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "af93018384416d411f6e12a70ba6282c"
server_key_iv: "e7b2bb61a63895fe2746410b08e19c68"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.753023 22779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:01:19.755510 22796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:19.756706 22779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:01:19.756855 22779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/wal
uuid: "0ed36f5fe31e4b8b8bae557261a709cd"
format_stamp: "Formatted at 2026-09-29 01:01:19 on dist-test-slave-30s6"
server_key: "af93018384416d411f6e12a70ba6282c"
server_key_iv: "e7b2bb61a63895fe2746410b08e19c68"
server_key_version: "encryptionkey@0"
I20260929 01:01:19.756965 22779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:19.768204 22779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:19.770751 22779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:19.770941 22779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:19.777585 22779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39473
I20260929 01:01:19.777863 22848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39473 every 8 connection(s)
I20260929 01:01:19.778589 22779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
I20260929 01:01:19.781230 22849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:19.782279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 22779
I20260929 01:01:19.782415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643398448434-392-0/minicluster-data/master-2/wal/instance
I20260929 01:01:19.782689 392 external_mini_cluster.cc:1428] Setting key 85b92ba9ae6b476b3544388d218c0206
I20260929 01:01:19.785914 22849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:19.797036 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.787882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.0.98.62:36879)
0929 01:01:19.788012 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.788016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.788318 (+ 302us) server_negotiation.cc:407] Connection header received
0929 01:01:19.788524 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.788527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.788587 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.788673 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.789290 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.789778 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.790687 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.790911 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.791022 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.793615 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.793619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.793621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.793649 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.795311 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.795751 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.795757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.795759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.795839 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.796149 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.796154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.796156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.796338 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.796443 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.796770 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.796876 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:01:19.797184 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.787787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36879 (local address 127.0.0.1:36608)
0929 01:01:19.788196 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.788243 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.788412 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.788720 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.788724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.789050 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.789163 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.789169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.789899 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.789905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.790567 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.790576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.791054 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.791058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.791161 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.791677 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.791692 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.793479 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.795414 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.795421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.795425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.795641 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.795940 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.795943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.795945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.796044 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.796454 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.796461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.796569 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.796707 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.796877 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":163,"threads_started":1}
I20260929 01:01:19.797190 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.783992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.0.98.60:39473)
0929 01:01:19.784400 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.784419 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.784466 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:01:19.785297 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.785319 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.785732 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.786082 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.786893 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.788167 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.789177 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.789644 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.789808 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.792202 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.792208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.792223 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.792243 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.794178 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.794711 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.794717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.794723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.794855 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.795144 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.795148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.795150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.795609 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.795798 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.796060 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.796442 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":97,"threads_started":1}
I20260929 01:01:19.798447 22849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:19.807124 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.798889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39379 (local address 127.0.0.1:35226)
0929 01:01:19.798991 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.799008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.799081 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.799445 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.799448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.799628 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.799746 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.799753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.800799 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.800803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.801482 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.801490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.802088 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.802090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.802190 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.802652 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.802666 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.803773 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.805589 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.805594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.805598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.805849 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.806149 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.806152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.806154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.806278 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.806662 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.806665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.806769 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.806887 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.807008 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 01:01:19.807407 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.799025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.0.98.61:39379)
0929 01:01:19.799213 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.799218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.799233 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:01:19.799277 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.799280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.799330 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.799419 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.799850 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.800695 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.801614 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.801980 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.802088 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.803914 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.803919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.803924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.803974 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.805479 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.805948 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.805953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.805956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.806047 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.806384 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.806386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.806387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.806554 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.806636 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.806961 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.807081 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 01:01:19.808092 22849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:19.808607 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.799582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.0.98.62:36879)
0929 01:01:19.799687 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.799691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.799705 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:01:19.799763 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.799766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.799812 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.799882 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.800562 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.801228 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.802228 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.802537 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.802641 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.805141 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.805145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.805147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.805171 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.806818 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.807289 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.807295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.807300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.807384 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.807758 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.807761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.807763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.807975 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.808214 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.808320 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.808439 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:01:19.817349 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.808525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50870)
0929 01:01:19.808615 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.808627 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.808718 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.809144 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.809146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.809263 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.809382 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.809389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.810750 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.810753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.811463 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.811472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.811938 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.811941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.811996 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.812440 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.812455 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.813855 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.815863 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.815865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.815866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.816128 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.816411 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.816414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.816416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.816516 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.816861 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.816863 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.816946 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.817053 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.817181 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 01:01:19.817353 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.808840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.0.98.60:39473)
0929 01:01:19.808931 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.808936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.808955 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:01:19.808982 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.808984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.809014 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.809114 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.809470 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.810640 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.811573 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.811836 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.811935 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.814002 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.814008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.814013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.814042 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.815521 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.816231 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.816233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.816235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.816304 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.816604 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.816611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.816612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.816769 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.816872 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.817096 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.817237 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":30}
I20260929 01:01:19.818955 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.809904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.0.98.61:39379)
0929 01:01:19.810025 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.810029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.810089 (+ 60us) server_negotiation.cc:407] Connection header received
0929 01:01:19.810241 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.810244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.810281 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.810351 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.811015 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.811689 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.812454 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.812810 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.812924 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.815639 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.815642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.815644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.815664 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.817091 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.817779 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.817781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.817783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.817851 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.818145 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.818147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.818148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.818351 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.818557 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.818693 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.818806 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 01:01:19.820842 22849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd: Bootstrap starting.
I20260929 01:01:19.822158 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:19.823559 22849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:19.824436 22849 log.cc:824] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:19.823518 22672 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "49ecb16605c542c5bbc0066d4f00a4df"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36618
I20260929 01:01:19.826618 22849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd: No bootstrap required, opened a new log
I20260929 01:01:19.827457 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } attempt: 3
I20260929 01:01:19.830041 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } attempt: 3
I20260929 01:01:19.830576 22849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } }
I20260929 01:01:19.830865 22849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed36f5fe31e4b8b8bae557261a709cd, State: Initialized, Role: FOLLOWER
I20260929 01:01:19.831555 22849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } }
I20260929 01:01:19.832048 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.832264 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:01:19.833061 22849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 01:01:19 dist-test-slave-30s6 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:01:19.836746 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.827898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50882)
0929 01:01:19.828054 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.828071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.828160 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.828500 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.828503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.828695 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.828777 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.828781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.829497 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.829501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.830082 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.830092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.830842 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.830846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.830942 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.831464 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.831478 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.832942 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.835049 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.835053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.835055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.835339 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.835611 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.835614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.835616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.835722 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.836193 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.836197 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.836316 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.836419 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.836575 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 01:01:19.836745 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.828174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.0.98.60:39473)
0929 01:01:19.828303 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.828309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.828326 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:01:19.828358 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.828360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.828397 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.828486 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.828898 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.829387 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.830298 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.830610 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.830716 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.833105 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.833111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.833114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.833139 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.834911 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.835426 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.835433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.835434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.835520 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.835825 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.835828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.835830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.836076 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.836191 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.836482 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.836591 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:01:19.840283 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728: Bootstrap starting.
I20260929 01:01:19.841310 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.830499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39473 (local address 127.0.0.1:50890)
0929 01:01:19.830687 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.830702 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:19.830783 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:19.831238 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:19.831241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:19.831458 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:19.831613 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.831622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.832925 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.832931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.833686 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:19.833697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.834542 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.834545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:19.834650 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.835099 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:19.835118 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:19.836689 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:19.839695 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.839698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.839701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.839947 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.840221 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:19.840224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:19.840226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:19.840325 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:19.840799 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:19.840803 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:19.840916 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 01:01:19.841009 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 01:01:19.841148 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 01:01:19.841385 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.830531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.0.98.60:39473)
0929 01:01:19.831039 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.831047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.831058 (+ 11us) server_negotiation.cc:407] Connection header received
0929 01:01:19.831085 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.831087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.831119 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.831216 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.831742 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.832826 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.833821 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.834418 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.834538 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.836832 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.836836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.836839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.836862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.839595 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.840038 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.840041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.840043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.840136 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.840432 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.840436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.840440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.840635 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.840750 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.841122 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.841233 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":111,"threads_started":1}
I20260929 01:01:19.838531 22779 master_runner.cc:186] Error getting master registration for 127.0.98.62:36879: OK, instance_id { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" instance_seqno: 1790643679365851 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:01:19.842595 22779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:01:19.843034 22773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728: Neither blocks nor log segments found. Creating new log.
W20260929 01:01:19.843298 22867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:19.843366 22867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:19.843900 22773 log.cc:824] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:19.844211 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df: Bootstrap starting.
I20260929 01:01:19.845726 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728: No bootstrap required, opened a new log
I20260929 01:01:19.846730 22698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:19.847580 22698 log.cc:824] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:19.849781 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df: No bootstrap required, opened a new log
I20260929 01:01:19.849855 22773 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } }
I20260929 01:01:19.850075 22773 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa53b1e0a7824e09932ce413c1dff728, State: Initialized, Role: FOLLOWER
I20260929 01:01:19.850445 22773 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } }
I20260929 01:01:19.850939 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.851207 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:19.851778 22773 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:19.852670 22698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } }
I20260929 01:01:19.852880 22698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ecb16605c542c5bbc0066d4f00a4df, State: Initialized, Role: FOLLOWER
I20260929 01:01:19.853173 22698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } }
I20260929 01:01:19.853695 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.853994 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:19.854607 22698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:19.854713 22672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 01:01:19.854820 22672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:19.856060 22672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } }
I20260929 01:01:19.857717 22747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49ecb16605c542c5bbc0066d4f00a4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fa53b1e0a7824e09932ce413c1dff728"
I20260929 01:01:19.857941 22747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:19.855608 22705 master_runner.cc:186] Error getting master registration for 127.0.98.62:36879: OK, instance_id { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" instance_seqno: 1790643679365851 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:01:19.858928 22705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:01:19.859171 22747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ecb16605c542c5bbc0066d4f00a4df in term 1.
I20260929 01:01:19.859599 22823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49ecb16605c542c5bbc0066d4f00a4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0ed36f5fe31e4b8b8bae557261a709cd"
I20260929 01:01:19.859824 22823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:19.861133 22823 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ecb16605c542c5bbc0066d4f00a4df in term 1.
I20260929 01:01:19.856933 22672 leader_election.cc:288] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [CANDIDATE]: Term 1 election: Requested vote from peers fa53b1e0a7824e09932ce413c1dff728 (127.0.98.61:39379), 0ed36f5fe31e4b8b8bae557261a709cd (127.0.98.60:39473)
I20260929 01:01:19.859565 22649 leader_election.cc:302] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ecb16605c542c5bbc0066d4f00a4df, fa53b1e0a7824e09932ce413c1dff728; no voters:
W20260929 01:01:19.861917 22894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:19.862143 22894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:01:19.862447 22651 tablet.cc:2369] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:01:19.863086 22877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:19.863265 22877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [term 1 LEADER]: Becoming Leader. State: Replica: 49ecb16605c542c5bbc0066d4f00a4df, State: Running, Role: LEADER
I20260929 01:01:19.863575 22877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } }
I20260929 01:01:19.866741 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ecb16605c542c5bbc0066d4f00a4df. Latest consensus state: current_term: 1 leader_uuid: "49ecb16605c542c5bbc0066d4f00a4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.866910 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:19.867324 22904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:19.868176 22904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:19.871462 22630 master_runner.cc:263] No leader master found from master 49ecb16605c542c5bbc0066d4f00a4df
I20260929 01:01:19.871534 22630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:01:19.874596 22883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 01:01:19.880098 22823 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [term 1 FOLLOWER]: Refusing update from remote peer 49ecb16605c542c5bbc0066d4f00a4df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:19.880785 22877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:19.885809 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ecb16605c542c5bbc0066d4f00a4df. Latest consensus state: current_term: 1 leader_uuid: "49ecb16605c542c5bbc0066d4f00a4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.885946 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:19.886168 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.864949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.0.98.60:39473)
0929 01:01:19.865052 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.865056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.870864 (+ 5808us) server_negotiation.cc:407] Connection header received
0929 01:01:19.871025 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.871028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.871086 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.871135 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.872108 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.872913 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.874090 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.874461 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.874536 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.882287 (+ 7751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.882291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.882294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.882322 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.884207 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.884740 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.884744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.884746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.884830 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.885125 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.885128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.885129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.885350 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.885477 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.885903 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.886027 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:01:19.879281 22747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [term 1 FOLLOWER]: Refusing update from remote peer 49ecb16605c542c5bbc0066d4f00a4df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:19.890848 22896 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:19.893896 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.864322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.0.98.61:39379)
0929 01:01:19.865302 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.865307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.871000 (+ 5693us) server_negotiation.cc:407] Connection header received
0929 01:01:19.871045 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.871048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.871093 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.871171 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.875511 (+ 4340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.876417 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.877683 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.878024 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.878093 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.889381 (+ 11288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.889385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.889388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.889416 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.891120 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.892499 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.892503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.892504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.892602 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.892980 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.892983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.892985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.893187 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.893294 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.893630 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.893744 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903}
I20260929 01:01:19.894255 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ecb16605c542c5bbc0066d4f00a4df. Latest consensus state: current_term: 1 leader_uuid: "49ecb16605c542c5bbc0066d4f00a4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.894371 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:19.895413 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49ecb16605c542c5bbc0066d4f00a4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.895516 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:19.897351 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49ecb16605c542c5bbc0066d4f00a4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.897459 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:19.897917 22905 mvcc.cc:205] Tried to move back new op lower bound from 7334476512779341824 to 7334476512724303872. Current Snapshot: MvccSnapshot[applied={T|T < 7334476512779341824}]
W20260929 01:01:19.898061 22883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:19.899963 22904 catalog_manager.cc:1347] Generated new cluster ID: e9aad366046c41bf8b0dca1ce3e055e8
I20260929 01:01:19.900029 22904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:19.903434 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49ecb16605c542c5bbc0066d4f00a4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.903544 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:19.904685 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49ecb16605c542c5bbc0066d4f00a4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ecb16605c542c5bbc0066d4f00a4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } peers { permanent_uuid: "fa53b1e0a7824e09932ce413c1dff728" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39379 } } peers { permanent_uuid: "0ed36f5fe31e4b8b8bae557261a709cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39473 } } }
I20260929 01:01:19.904789 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:19.905508 22910 mvcc.cc:205] Tried to move back new op lower bound from 7334476512779341824 to 7334476512724303872. Current Snapshot: MvccSnapshot[applied={T|T < 7334476512779341824}]
I20260929 01:01:19.911007 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.870529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.0.98.62:36879)
0929 01:01:19.870636 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.870642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.882628 (+ 11986us) server_negotiation.cc:407] Connection header received
0929 01:01:19.882795 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.882798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.882843 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.882891 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.886401 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.887173 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.889203 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:19.889536 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:19.889616 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:19.897024 (+ 7408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:19.897028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:19.897032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:19.897062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:19.898738 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.899200 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.899204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.899206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.899284 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:19.899551 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:19.899554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:19.899556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:19.899760 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:19.899865 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:19.910693 (+ 10828us) server_negotiation.cc:299] Negotiation successful
0929 01:01:19.910832 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:01:19.912814 22908 mvcc.cc:205] Tried to move back new op lower bound from 7334476512779341824 to 7334476512724303872. Current Snapshot: MvccSnapshot[applied={T|T < 7334476512779341824}]
I20260929 01:01:19.945557 22904 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:19.947144 22904 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:19.965852 22904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ecb16605c542c5bbc0066d4f00a4df: Generated new TSK 0
I20260929 01:01:19.966841 22904 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:01:19.996156 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:19.992646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.0.98.61:39379)
0929 01:01:19.992937 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.992943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.994844 (+ 1901us) server_negotiation.cc:407] Connection header received
0929 01:01:19.995251 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.995255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.995309 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.995365 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.995996 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35246: BlockingRecv error: recv got EOF from 127.0.0.1:35246 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 01:01:19.997514 22855 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:19.993591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.0.98.60:39473)
0929 01:01:19.993728 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.993733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.995163 (+ 1430us) server_negotiation.cc:407] Connection header received
0929 01:01:19.996772 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.996775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.996822 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.996871 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.997399 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50902: BlockingRecv error: recv got EOF from 127.0.0.1:50902 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 01:01:19.998643 22702 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:19.994213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.0.98.62:36879)
0929 01:01:19.994329 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:19.994333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:19.996587 (+ 2254us) server_negotiation.cc:407] Connection header received
0929 01:01:19.997868 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:19.997871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:19.997921 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:19.997971 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:19.998512 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36640: BlockingRecv error: recv got EOF from 127.0.0.1:36640 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 01:01:20.003468 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.001833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.0.98.61:39379)
0929 01:01:20.001937 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.001943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.002491 (+ 548us) server_negotiation.cc:407] Connection header received
0929 01:01:20.002726 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.002729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.002809 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.002867 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.003366 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35248: BlockingRecv error: recv got EOF from 127.0.0.1:35248 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 01:01:20.008422 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.006862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.0.98.61:39379)
0929 01:01:20.006965 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.006969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.007490 (+ 521us) server_negotiation.cc:407] Connection header received
0929 01:01:20.007699 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.007702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.007754 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.007810 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.008319 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35262: BlockingRecv error: recv got EOF from 127.0.0.1:35262 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 01:01:20.013934 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.012371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.0.98.61:39379)
0929 01:01:20.012475 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.012479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.013018 (+ 539us) server_negotiation.cc:407] Connection header received
0929 01:01:20.013214 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.013218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.013269 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.013323 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.013837 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35268: BlockingRecv error: recv got EOF from 127.0.0.1:35268 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 01:01:20.021512 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.019899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.0.98.61:39379)
0929 01:01:20.020041 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.020045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.020593 (+ 548us) server_negotiation.cc:407] Connection header received
0929 01:01:20.020791 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.020794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.020846 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.020901 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.021410 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35282: BlockingRecv error: recv got EOF from 127.0.0.1:35282 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 01:01:20.033238 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.031657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.0.98.61:39379)
0929 01:01:20.031777 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.031781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.032306 (+ 525us) server_negotiation.cc:407] Connection header received
0929 01:01:20.032493 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.032497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.032544 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.032607 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.033123 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35286: BlockingRecv error: recv got EOF from 127.0.0.1:35286 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 01:01:20.052618 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.051382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.0.98.61:39379)
0929 01:01:20.051511 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.051516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.051749 (+ 233us) server_negotiation.cc:407] Connection header received
0929 01:01:20.051906 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.051909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.051971 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.052045 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.052483 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35298: BlockingRecv error: recv got EOF from 127.0.0.1:35298 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 01:01:20.087926 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.086753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.0.98.61:39379)
0929 01:01:20.086904 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.086909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.087047 (+ 138us) server_negotiation.cc:407] Connection header received
0929 01:01:20.087147 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.087150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.087211 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.087320 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.087791 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35312: BlockingRecv error: recv got EOF from 127.0.0.1:35312 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 01:01:20.155345 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.154060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.0.98.61:39379)
0929 01:01:20.154247 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.154253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.154436 (+ 183us) server_negotiation.cc:407] Connection header received
0929 01:01:20.154632 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.154635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.154695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.154786 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.155230 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35318: BlockingRecv error: recv got EOF from 127.0.0.1:35318 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 01:01:20.286769 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.285516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.0.98.61:39379)
0929 01:01:20.285687 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.285692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.285807 (+ 115us) server_negotiation.cc:407] Connection header received
0929 01:01:20.285932 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.285934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.285988 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.286061 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.286601 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35322: BlockingRecv error: recv got EOF from 127.0.0.1:35322 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 01:01:20.545874 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:20.544750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.0.98.61:39379)
0929 01:01:20.544907 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.544911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.545026 (+ 115us) server_negotiation.cc:407] Connection header received
0929 01:01:20.545161 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.545163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.545210 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.545284 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.545726 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35328: BlockingRecv error: recv got EOF from 127.0.0.1:35328 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:01:20.844770 22867 catalog_manager.cc:1259] Loaded cluster ID: e9aad366046c41bf8b0dca1ce3e055e8
I20260929 01:01:20.844868 22867 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd: loading cluster ID for follower catalog manager: success
I20260929 01:01:20.846746 22867 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd: acquiring CA information for follower catalog manager: success
I20260929 01:01:20.848060 22867 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ed36f5fe31e4b8b8bae557261a709cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:01:20.899462 22883 catalog_manager.cc:1259] Loaded cluster ID: e9aad366046c41bf8b0dca1ce3e055e8
I20260929 01:01:20.899588 22883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728: loading cluster ID for follower catalog manager: success
I20260929 01:01:20.901232 22883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728: acquiring CA information for follower catalog manager: success
I20260929 01:01:20.902405 22883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa53b1e0a7824e09932ce413c1dff728: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:01:21.064002 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:21.059808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.0.98.61:39379)
0929 01:01:21.060063 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:21.060068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:21.060208 (+ 140us) server_negotiation.cc:407] Connection header received
0929 01:01:21.060380 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:21.060383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:21.060537 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:21.060662 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:21.060939 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.061469 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.062438 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.062676 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.062789 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:21.063120 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:21.063602 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:21.063742 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:21.063779 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 01:01:21.063838 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260929 01:01:21.070768 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:21.067125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.0.98.60:39473)
0929 01:01:21.067382 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:21.067386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:21.067400 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:01:21.067495 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:21.067498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:21.067610 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:21.067710 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:21.067883 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.068377 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.069172 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.069439 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.069567 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:21.069799 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:21.070407 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:21.070550 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:21.070605 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 01:01:21.070652 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260929 01:01:21.077590 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:21.074102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.98.60:39473)
0929 01:01:21.074268 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:21.074272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:21.074576 (+ 304us) server_negotiation.cc:407] Connection header received
0929 01:01:21.074716 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:21.074719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:21.074763 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:21.074829 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:21.075077 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.075675 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.076495 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.076705 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.076818 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:21.077127 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:21.077218 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:21.077291 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:21.077412 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 01:01:21.077463 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:01:21.077956 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:21.073557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.0.98.61:39379)
0929 01:01:21.073682 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:21.073686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:21.073920 (+ 234us) server_negotiation.cc:407] Connection header received
0929 01:01:21.074083 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:21.074086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:21.074168 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:21.074252 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:21.074596 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.075345 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.076771 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.077092 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.077180 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:21.077476 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:21.077547 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:21.077592 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:21.077776 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 01:01:21.077828 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:01:21.078888 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:21.073508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.0.98.62:36879)
0929 01:01:21.073748 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:21.073752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:21.073827 (+ 75us) server_negotiation.cc:407] Connection header received
0929 01:01:21.074016 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:21.074019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:21.074211 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:21.074328 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:21.074669 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.075367 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.077236 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:21.077596 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:21.077697 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:21.078208 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:21.078520 (+ 312us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:21.078616 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:21.078679 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 01:01:21.078736 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260929 01:01:21.082058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 22630
I20260929 01:01:21.088531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 22705
I20260929 01:01:21.094133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 22779
2026-09-29T01:01:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3069 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:21 dist-test-slave-30s6 krb5kdc[22997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:21 dist-test-slave-30s6 krb5kdc[22997](info): set up 2 sockets
Sep 29 01:01:21 dist-test-slave-30s6 krb5kdc[22997](info): commencing operation
krb5kdc: starting...
W20260929 01:01:22.122625 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/test-user.keytab.
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:01:22.245281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40393
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40393,127.0.98.61:41127,127.0.98.60:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:22.354251 23010 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:01:22.354519 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:22.354542 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:22.354624 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:22.358639 23010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:22.358690 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:22.358711 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:22.358726 23010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:22.358774 23010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:22.363871 23010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40393,127.0.98.61:41127,127.0.98.60:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:22.364846 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:22.365871 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:22.374312 23015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:22.374338 23018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:22.374313 23016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:22.374629 23010 server_base.cc:1034] running on GCE node
I20260929 01:01:22.375146 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:22.376194 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:22.377377 23010 hybrid_clock.cc:648] HybridClock initialized: now 1790643682377353 us; error 33 us; skew 500 ppm
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:22.380180 23010 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:01:22.381080 23010 webserver.cc:460] Webserver started at http://127.0.98.62:43665/ using document root <none> and password file <none>
I20260929 01:01:22.381610 23010 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:22.381669 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:22.381886 23010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:22.383888 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/data/instance:
uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "1cf29e7f29d8a33f75b79aa74abf0d2c"
server_key_iv: "86076e3bf4acf39a2c768e63456e51fe"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.384440 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "1cf29e7f29d8a33f75b79aa74abf0d2c"
server_key_iv: "86076e3bf4acf39a2c768e63456e51fe"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.388051 23010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:01:22.390386 23025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:22.391580 23010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:01:22.391710 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/wal
uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "1cf29e7f29d8a33f75b79aa74abf0d2c"
server_key_iv: "86076e3bf4acf39a2c768e63456e51fe"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.391811 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:22.406997 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:22.409964 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:22.410162 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:22.417241 23010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40393
I20260929 01:01:22.417532 23077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40393 every 8 connection(s)
I20260929 01:01:22.418298 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:22.421029 23078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:22.421198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 23010
I20260929 01:01:22.421325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-0/wal/instance
I20260929 01:01:22.421640 392 external_mini_cluster.cc:1428] Setting key 36d8b45503f289155f9db08d60952706
I20260929 01:01:22.425305 23078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:22.441303 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.427659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40393 (local address 127.0.0.1:52940)
0929 01:01:22.428191 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.428226 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.429053 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.429769 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.429774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.430246 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.430611 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.430625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.431895 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.431897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.432962 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.432977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.433649 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.433652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.433758 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.434390 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.434406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.435847 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.438501 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.438508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.438517 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.438747 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.439132 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.439134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.439135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.439238 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.439991 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.439996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.440119 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.440255 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.440510 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":3,"spinlock_wait_cycles":18816,"thread_start_us":168,"threads_started":1}
I20260929 01:01:22.441306 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.427658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.0.98.62:40393)
0929 01:01:22.428059 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.428078 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.428332 (+ 254us) server_negotiation.cc:407] Connection header received
0929 01:01:22.429354 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.429358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.429593 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.429776 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.430754 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.431740 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.433107 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.433475 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.433691 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.436012 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.436021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.436037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.436071 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.438361 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.438871 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.438879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.438882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.439029 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.439331 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.439335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.439338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.439877 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.440033 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.440380 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.440589 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260929 01:01:22.443840 23078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:22.444550 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.444309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33272)
0929 01:01:22.444414 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.444477 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 01:01:22.444762 23026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
I20260929 01:01:22.445658 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.445454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33286)
0929 01:01:22.445568 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.445611 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 01:01:22.445791 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41127: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 01:01:22.449236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41127
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40393,127.0.98.61:41127,127.0.98.60:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:22.495047 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } attempt: 1
I20260929 01:01:22.496431 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.495673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33294)
0929 01:01:22.496257 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.496332 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 01:01:22.497596 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.497307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33304)
0929 01:01:22.497481 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.497541 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 01:01:22.497752 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41127: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
W20260929 01:01:22.558125 23085 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:01:22.558496 23085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:22.558571 23085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:22.558653 23085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:22.562659 23085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:22.562745 23085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:22.562783 23085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:22.562820 23085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:22.562860 23085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:22.565928 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } attempt: 2
I20260929 01:01:22.566787 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.566490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33306)
0929 01:01:22.566647 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.566710 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 01:01:22.567488 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.567281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33310)
0929 01:01:22.567387 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.567438 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:01:22.567615 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41127: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41127: connect: Connection refused (error 111)
I20260929 01:01:22.568625 23085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40393,127.0.98.61:41127,127.0.98.60:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:22.569671 23085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:22.570873 23085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:22.579166 23093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:22.579236 23090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:22.579329 23091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:22.579376 23085 server_base.cc:1034] running on GCE node
I20260929 01:01:22.579950 23085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:22.580993 23085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:22.582170 23085 hybrid_clock.cc:648] HybridClock initialized: now 1790643682582124 us; error 55 us; skew 500 ppm
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:22.585278 23085 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 01:01:22.586261 23085 webserver.cc:460] Webserver started at http://127.0.98.61:41245/ using document root <none> and password file <none>
I20260929 01:01:22.586791 23085 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:22.586855 23085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:22.587023 23085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:22.589035 23085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/data/instance:
uuid: "f2e2479154ef42998e902e263d867714"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "751127239195f4995846640384d0d1a6"
server_key_iv: "0332cb2743fb6d0506e6f7fe87ed8540"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.589565 23085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/wal/instance:
uuid: "f2e2479154ef42998e902e263d867714"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "751127239195f4995846640384d0d1a6"
server_key_iv: "0332cb2743fb6d0506e6f7fe87ed8540"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.593369 23085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:01:22.596021 23100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:22.597319 23085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:01:22.597532 23085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/wal
uuid: "f2e2479154ef42998e902e263d867714"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "751127239195f4995846640384d0d1a6"
server_key_iv: "0332cb2743fb6d0506e6f7fe87ed8540"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.597654 23085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:22.617609 23085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:22.620754 23085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:22.620988 23085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:22.628396 23085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41127
I20260929 01:01:22.628677 23152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41127 every 8 connection(s)
I20260929 01:01:22.629437 23085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/data/info.pb
I20260929 01:01:22.632370 23153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:22.635124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 23085
I20260929 01:01:22.635285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-1/wal/instance
I20260929 01:01:22.635610 392 external_mini_cluster.cc:1428] Setting key 5f3b0d09bbbfdeb3726c4e29aefafb8c
I20260929 01:01:22.636202 23153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:22.651183 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.638069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.0.98.62:40393)
0929 01:01:22.638241 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.638247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.639156 (+ 909us) server_negotiation.cc:407] Connection header received
0929 01:01:22.640065 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.640072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.640140 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.640247 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.641373 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.642273 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.643529 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.643782 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.643913 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.647080 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.647087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.647093 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.647130 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.648766 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.649401 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.649405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.649406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.649495 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.649906 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.649909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.649910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.650163 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.650287 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.650954 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.651060 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:01:22.652395 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.638516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40393 (local address 127.0.0.1:52950)
0929 01:01:22.638982 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.639033 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.639850 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.640398 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.640404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.640751 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.641214 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.641226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.642471 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.642474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.643385 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.643405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.643939 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.643944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.644129 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.644716 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.644733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.646913 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.648882 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.648890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.648908 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.649278 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.649606 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.649612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.649617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.649810 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.650299 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.650308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.650664 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.650886 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.651257 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":139,"threads_started":1}
I20260929 01:01:22.653863 23153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:22.658772 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } attempt: 3
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 01:01:22.661576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIUjItr/build/debug/bin/kudu
/tmp/dist-test-taskIUjItr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46489
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40393,127.0.98.61:41127,127.0.98.60:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:22.663874 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.654368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33320)
0929 01:01:22.654520 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.654535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.654608 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.655201 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.655203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.655335 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.655417 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.655422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.656572 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.656575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.657141 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.657149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.657825 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.657828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.657896 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.658388 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.658399 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.659905 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.662108 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.662114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.662118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.662445 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.662760 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.662763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.662765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.662838 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.663371 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.663374 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.663458 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.663590 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.663709 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 01:01:22.664186 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.654527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.0.98.61:41127)
0929 01:01:22.654850 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.654858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.654874 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:22.654904 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.654908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.655071 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.655205 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.655541 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.656459 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.657241 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.657729 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.657814 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.660043 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.660056 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.660070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.660091 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.661987 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.662566 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.662575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.662577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.662670 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.662940 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.662943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.662945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.663267 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.663398 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.663668 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.663790 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":136,"threads_started":1}
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:01:22.665982 23153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:22.666671 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.666436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52674)
0929 01:01:22.666541 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.666611 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 01:01:22.666855 23101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
I20260929 01:01:22.667635 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.667434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52680)
0929 01:01:22.667534 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.667579 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 01:01:22.667852 23153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
I20260929 01:01:22.668841 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.659226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33324)
0929 01:01:22.659345 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.659367 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.659465 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.659839 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.659842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.660036 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.660176 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.660182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.661246 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.661252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.662138 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.662146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.662714 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.662721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.662860 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.663330 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.663347 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.664896 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.667212 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.667217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.667220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.667506 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.667830 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.667835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.667837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.667936 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.668331 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.668334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.668435 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.668553 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.668690 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 01:01:22.668869 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.659247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.98.61:41127)
0929 01:01:22.659579 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.659587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.659605 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:01:22.659657 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.659680 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.659723 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.659823 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.660283 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.661140 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.662282 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.662594 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.662709 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.665064 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.665071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.665077 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.665106 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.667079 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.667645 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.667650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.667651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.667739 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.668069 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.668072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.668073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.668223 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.668339 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.668604 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.668705 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260929 01:01:22.669596 23078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:22.670337 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.670025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52696)
0929 01:01:22.670165 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.670252 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 01:01:22.671051 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.670753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52704)
0929 01:01:22.670914 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.670977 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 01:01:22.671185 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
I20260929 01:01:22.702338 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } attempt: 1
I20260929 01:01:22.703218 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.702891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52718)
0929 01:01:22.703054 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.703127 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 01:01:22.704020 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.703789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52722)
0929 01:01:22.703903 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.703953 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 01:01:22.704192 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
I20260929 01:01:22.717108 23153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } attempt: 1
I20260929 01:01:22.717948 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.717678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52726)
0929 01:01:22.717807 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.717876 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 01:01:22.718729 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.718540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52728)
0929 01:01:22.718637 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.718681 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:01:22.718865 23153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
W20260929 01:01:22.774734 23160 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:01:22.775018 23160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:22.775043 23160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:22.775080 23160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:22.779352 23160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:22.779417 23160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260929 01:01:22.779345 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } attempt: 2
W20260929 01:01:22.779445 23160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:22.779469 23160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:22.779495 23160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:22.780141 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.779826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52736)
0929 01:01:22.779984 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.780052 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 01:01:22.780875 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.780664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52740)
0929 01:01:22.780770 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.780818 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:01:22.781045 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
I20260929 01:01:22.787025 23153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } attempt: 2
I20260929 01:01:22.787804 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.787471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52744)
0929 01:01:22.787640 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.787702 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 01:01:22.787887 23160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40393,127.0.98.61:41127,127.0.98.60:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:01:22.788457 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.788284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52746)
0929 01:01:22.788362 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.788405 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 01:01:22.788607 23153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46489: connect: Connection refused (error 111)
I20260929 01:01:22.789134 23160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:22.790148 23160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:22.798301 23167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:22.798363 23169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:22.798466 23160 server_base.cc:1034] running on GCE node
W20260929 01:01:22.798296 23166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:22.799136 23160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:22.800258 23160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:22.801440 23160 hybrid_clock.cc:648] HybridClock initialized: now 1790643682801418 us; error 30 us; skew 500 ppm
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:22.804132 23160 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 01:01:22.804971 23160 webserver.cc:460] Webserver started at http://127.0.98.60:41429/ using document root <none> and password file <none>
I20260929 01:01:22.805478 23160 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:22.805536 23160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:22.805694 23160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:22.807693 23160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/data/instance:
uuid: "a9adb78cacc24913b4e65600fe4db7c9"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "ba1338d3734ef7566291becba48eefd3"
server_key_iv: "bcc94536295c075f5bfd099ea7a72b42"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.808269 23160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/wal/instance:
uuid: "a9adb78cacc24913b4e65600fe4db7c9"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "ba1338d3734ef7566291becba48eefd3"
server_key_iv: "bcc94536295c075f5bfd099ea7a72b42"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.811800 23160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:01:22.814131 23176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:22.815351 23160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:01:22.815496 23160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/wal
uuid: "a9adb78cacc24913b4e65600fe4db7c9"
format_stamp: "Formatted at 2026-09-29 01:01:22 on dist-test-slave-30s6"
server_key: "ba1338d3734ef7566291becba48eefd3"
server_key_iv: "bcc94536295c075f5bfd099ea7a72b42"
server_key_version: "encryptionkey@0"
I20260929 01:01:22.815599 23160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:22.848035 23160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:22.851457 23160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:22.851611 23160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:22.858596 23160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46489
I20260929 01:01:22.858876 23228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46489 every 8 connection(s)
I20260929 01:01:22.859625 23160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/data/info.pb
I20260929 01:01:22.862350 23229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:22.865957 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:22.869109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIUjItr/build/debug/bin/kudu as pid 23160
I20260929 01:01:22.869239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIUjItr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643398448434-392-0/minicluster-data/master-2/wal/instance
I20260929 01:01:22.869527 392 external_mini_cluster.cc:1428] Setting key 903912f95964dd7c48bb94e18ea4c5f9
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:01:22.879776 23153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } attempt: 3
I20260929 01:01:22.880429 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } attempt: 3
I20260929 01:01:22.881340 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.867769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.0.98.62:40393)
0929 01:01:22.867938 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.867943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.868764 (+ 821us) server_negotiation.cc:407] Connection header received
0929 01:01:22.869925 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.869930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.870001 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.870100 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.871710 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.872455 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.873764 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.873993 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.874095 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.876839 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.876843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.876845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.876873 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.878785 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.879448 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.879450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.879451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.879523 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.880045 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.880048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.880050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.880282 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.880390 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.881046 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.881165 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:01:22.882841 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.868135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40393 (local address 127.0.0.1:52966)
0929 01:01:22.868602 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.868648 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.869682 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.870334 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.870344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.870845 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.871545 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.871568 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.872588 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.872590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.873632 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.873644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.874131 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.874135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.874403 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.874991 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.875009 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.876667 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.878913 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.878921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.878930 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.879341 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.879617 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.879621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.879626 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.879816 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.880455 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.880463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.880757 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.881550 (+ 793us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.881858 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":63,"spinlock_wait_cycles":2816,"thread_start_us":96,"threads_started":1}
I20260929 01:01:22.882844 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.870148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.0.98.60:46489)
0929 01:01:22.870531 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.870543 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.870562 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:01:22.870695 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.870699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.870840 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.870968 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.871917 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.872957 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.874024 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.874510 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.874589 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.877322 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.877330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.877344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.877372 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.879522 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.879933 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.879943 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.879944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.880040 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.880593 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.880596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.880598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.880869 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.881067 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.881699 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.881905 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":296,"spinlock_wait_cycles":1920,"thread_start_us":163,"threads_started":1}
I20260929 01:01:22.884050 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:01:22.891903 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.880270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52760)
0929 01:01:22.880418 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.880434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.880505 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.881302 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.881305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.881531 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.881655 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.881665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.882571 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.882576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.883450 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.883465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.884120 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.884125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.884222 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.884701 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.884720 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.886154 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.890203 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.890210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.890215 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.890476 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.890821 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.890826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.890829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.890911 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.891387 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.891393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.891494 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.891575 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.891710 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 01:01:22.892522 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.880723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.0.98.60:46489)
0929 01:01:22.881041 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.881053 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.881068 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:01:22.881105 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.881108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.881150 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.881778 (+ 628us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.881851 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.882432 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.883608 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.883980 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.884466 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.886327 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.886330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.886333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.886354 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.890051 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.890612 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.890616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.890619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.890708 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.891048 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.891054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.891058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.891264 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.892091 (+ 827us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.892164 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.892299 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260929 01:01:22.894613 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.880837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52768)
0929 01:01:22.881029 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.881044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.881120 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.882598 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.882601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.882772 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.882903 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.882911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.885105 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.885109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.885960 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.885974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.886712 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.886718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.886812 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.887240 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.887254 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.888630 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.891971 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.891978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.891982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.892840 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.893480 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.893483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.893485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.893576 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.894001 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.894005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.894124 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.894324 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.894454 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260929 01:01:22.895089 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.885004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.98.61:41127)
0929 01:01:22.885122 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.885126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.885141 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:01:22.885178 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.885181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.885223 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.885305 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.886357 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.887348 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.888094 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.888405 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.888514 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.891314 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.891320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.891325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.891356 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.893026 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.893602 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.893609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.893611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.893704 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.894111 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.894115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.894116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.894389 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.894493 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.894828 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.894949 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:01:22.895419 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.881320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.0.98.60:46489)
0929 01:01:22.882359 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.882368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.882381 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:01:22.882415 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.882417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.882459 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.883583 (+ 1124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.883697 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.884793 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.886137 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.886546 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.886708 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.888763 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.888770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.888776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.889277 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.891488 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.893273 (+ 1785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.893276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.893278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.893358 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.893685 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.893691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.893693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.893894 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.893996 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.895071 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.895247 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":977,"spinlock_wait_cycles":2944,"thread_start_us":89,"threads_started":1}
I20260929 01:01:22.895717 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.884610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41127 (local address 127.0.0.1:33326)
0929 01:01:22.884726 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.884741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.884813 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.885435 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.885439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.885611 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.886224 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.886233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.887475 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.887477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.887968 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.887974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.888523 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.888527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.888631 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.889297 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.889315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.891140 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.893147 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.893152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.893155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.893459 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.893841 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.893844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.893849 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.893960 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.894537 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.894540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.894619 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.895219 (+ 600us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.895330 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"spinlock_wait_cycles":395264}
I20260929 01:01:22.896435 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:01:22.897001 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.885659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.0.98.62:40393)
0929 01:01:22.885748 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.885752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.885922 (+ 170us) server_negotiation.cc:407] Connection header received
0929 01:01:22.886079 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.886083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.886127 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.886199 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.887052 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.887777 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.888892 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.889106 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.889191 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.892163 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.892166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.892170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.892192 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.894092 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.894608 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.894612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.894613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.894707 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.895757 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.895761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.895763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.896047 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.896422 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.896721 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.896837 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 01:01:22.897127 23153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714: Bootstrap starting.
I20260929 01:01:22.899866 23078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea: Bootstrap starting.
I20260929 01:01:22.900665 23153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:22.901657 23153 log.cc:824] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:01:22.902729 23078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:22.903625 23078 log.cc:824] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:22.904104 23153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714: No bootstrap required, opened a new log
Sep 29 01:01:22 dist-test-slave-30s6 krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:01:22.905615 23078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea: No bootstrap required, opened a new log
I20260929 01:01:22.905732 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.896868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46489 (local address 127.0.0.1:52772)
0929 01:01:22.897151 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:22.897166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:22.897244 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:22.897505 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:22.897508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:22.897666 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:22.897740 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.897746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.898675 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.898679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.899416 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:22.899426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.900010 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.900013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:22.900084 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.900598 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:22.900612 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:22.902115 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:22.904222 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.904227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.904229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.904473 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.904718 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:22.904720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:22.904722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:22.904788 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:22.905187 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:22.905189 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:22.905265 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 01:01:22.905398 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 01:01:22.905528 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 01:01:22.906369 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.897048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.0.98.60:46489)
0929 01:01:22.897143 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.897150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.897233 (+ 83us) server_negotiation.cc:407] Connection header received
0929 01:01:22.897313 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.897318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.897354 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.897419 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.897848 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.898567 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.899536 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.899848 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.900264 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.902265 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.902268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.902271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.902293 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.904073 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.904567 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.904570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.904572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.904638 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.904865 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.904868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.904869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.905068 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.905155 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.906082 (+ 927us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.906228 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 01:01:22.909206 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9: Bootstrap starting.
I20260929 01:01:22.909221 23153 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } }
I20260929 01:01:22.909499 23153 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e2479154ef42998e902e263d867714, State: Initialized, Role: FOLLOWER
I20260929 01:01:22.909595 23078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } }
I20260929 01:01:22.909787 23078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f00c68dc0c1413393ca6fc99ef5c8ea, State: Initialized, Role: FOLLOWER
I20260929 01:01:22.909852 23153 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } }
I20260929 01:01:22.910094 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.898420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.0.98.61:41127)
0929 01:01:22.898539 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.898545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.898704 (+ 159us) server_negotiation.cc:407] Connection header received
0929 01:01:22.898879 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.898882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.898943 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.898996 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.900023 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.900756 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.902017 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.902359 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.902444 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.906060 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.906066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.906069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.906096 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.908156 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.908663 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.908666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.908668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.908753 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.909084 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.909087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.909089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.909297 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.909460 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.909826 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.909955 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 01:01:22.910051 23078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } }
I20260929 01:01:22.911315 23078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:22.911518 23153 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:01:22.911688 23031 tablet.cc:2369] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:01:22.912428 23229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:22.913359 23229 log.cc:824] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:22.914083 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:22.915601 23052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52970
I20260929 01:01:22.915802 23052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 01:01:22.915925 23052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:22.917207 23052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } }
W20260929 01:01:22.917796 23262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:22.917874 23262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:22.918134 23052 leader_election.cc:288] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [CANDIDATE]: Term 1 election: Requested vote from peers f2e2479154ef42998e902e263d867714 (127.0.98.61:41127), a9adb78cacc24913b4e65600fe4db7c9 (127.0.98.60:46489)
I20260929 01:01:22.918319 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.918455 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:22.918538 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.918689 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:22.919052 23127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2e2479154ef42998e902e263d867714"
I20260929 01:01:22.919276 23127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:22.920635 23127 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f00c68dc0c1413393ca6fc99ef5c8ea in term 1.
I20260929 01:01:22.921046 23026 leader_election.cc:302] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f00c68dc0c1413393ca6fc99ef5c8ea, f2e2479154ef42998e902e263d867714; no voters:
I20260929 01:01:22.921300 23239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:22.921525 23239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [term 1 LEADER]: Becoming Leader. State: Replica: 8f00c68dc0c1413393ca6fc99ef5c8ea, State: Running, Role: LEADER
I20260929 01:01:22.921875 23239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } }
I20260929 01:01:22.923756 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f00c68dc0c1413393ca6fc99ef5c8ea. Latest consensus state: current_term: 1 leader_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.923856 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: This master's current role is: LEADER
W20260929 01:01:22.924901 23255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:22.925168 23255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:22.925354 23267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:22.926299 23267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:22.923731 23085 master_runner.cc:186] Error getting master registration for 127.0.98.60:46489: OK, instance_id { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" instance_seqno: 1790643682856656 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:01:22.931870 23085 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:01:22.920816 23010 master_runner.cc:186] Error getting master registration for 127.0.98.60:46489: OK, instance_id { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" instance_seqno: 1790643682856656 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:01:22.932072 23010 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:01:22.936069 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9: No bootstrap required, opened a new log
I20260929 01:01:22.938812 23127 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [term 1 FOLLOWER]: Refusing update from remote peer 8f00c68dc0c1413393ca6fc99ef5c8ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:22.939602 23265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:22.940649 23229 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } }
I20260929 01:01:22.940893 23229 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9adb78cacc24913b4e65600fe4db7c9, State: Initialized, Role: FOLLOWER
I20260929 01:01:22.941216 23229 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } }
I20260929 01:01:22.942546 23229 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:22.943374 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f00c68dc0c1413393ca6fc99ef5c8ea. Latest consensus state: current_term: 1 leader_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.943498 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:22.943614 23203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9adb78cacc24913b4e65600fe4db7c9"
I20260929 01:01:22.943837 23203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:22.945262 23203 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f00c68dc0c1413393ca6fc99ef5c8ea in term 1.
I20260929 01:01:22.945861 23202 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [term 1 FOLLOWER]: Refusing update from remote peer 8f00c68dc0c1413393ca6fc99ef5c8ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:22.947023 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.947172 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:22.947026 23265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:22.953979 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f00c68dc0c1413393ca6fc99ef5c8ea. Latest consensus state: current_term: 1 leader_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.954105 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:22.957093 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.957222 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:22.958489 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.958624 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:22.958984 23274 mvcc.cc:205] Tried to move back new op lower bound from 7334476525304426496 to 7334476525251788800. Current Snapshot: MvccSnapshot[applied={T|T < 7334476525304426496}]
I20260929 01:01:22.959383 23277 mvcc.cc:205] Tried to move back new op lower bound from 7334476525304426496 to 7334476525251788800. Current Snapshot: MvccSnapshot[applied={T|T < 7334476525304426496}]
I20260929 01:01:22.960369 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.930261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.0.98.60:46489)
0929 01:01:22.930390 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.930405 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.933992 (+ 3587us) server_negotiation.cc:407] Connection header received
0929 01:01:22.934290 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.934293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.934346 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.934400 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.939769 (+ 5369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.940478 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.943866 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.944213 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.944288 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.950065 (+ 5777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.950069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.950072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.950098 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.951740 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.956729 (+ 4989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.956733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.956735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.956823 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.957581 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.957586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.957588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.957820 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.957981 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.960090 (+ 2109us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.960216 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:01:22.962080 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.927973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.0.98.61:41127)
0929 01:01:22.928111 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.928116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.933776 (+ 5660us) server_negotiation.cc:407] Connection header received
0929 01:01:22.934564 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.934567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.934618 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.934668 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.937862 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.938637 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.943319 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.943641 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.943722 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.952285 (+ 8563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.952291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.952295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.952328 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.953992 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.957171 (+ 3179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.957174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.957176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.957262 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.958741 (+ 1479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.958746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.958748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.958984 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.959101 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.961782 (+ 2681us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.961930 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:01:22.963683 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:22.933177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.0.98.62:40393)
0929 01:01:22.933275 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:22.933281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:22.936644 (+ 3363us) server_negotiation.cc:407] Connection header received
0929 01:01:22.936809 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:22.936813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:22.936860 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:22.936925 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:22.941528 (+ 4603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.942251 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.945517 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:22.945830 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:22.945925 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:22.954295 (+ 8370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:22.954299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:22.954301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:22.954328 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:22.955979 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.958062 (+ 2083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.958065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.958067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.958154 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:22.959404 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:22.959410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:22.959415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:22.959634 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:22.959739 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:22.963432 (+ 3693us) server_negotiation.cc:299] Negotiation successful
0929 01:01:22.963548 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 01:01:22.965600 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.965721 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:22.966130 23267 catalog_manager.cc:1347] Generated new cluster ID: d2557d29b9b94f2984d245a6b2b6abdc
I20260929 01:01:22.966941 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f00c68dc0c1413393ca6fc99ef5c8ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } peers { permanent_uuid: "f2e2479154ef42998e902e263d867714" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41127 } } peers { permanent_uuid: "a9adb78cacc24913b4e65600fe4db7c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46489 } } }
I20260929 01:01:22.967026 23267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:22.967054 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:01:22.968928 23289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:22.969008 23289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:23.015556 23267 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:23.018461 23267 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:23.031203 23267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f00c68dc0c1413393ca6fc99ef5c8ea: Generated new TSK 0
I20260929 01:01:23.032130 23267 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:01:23.066864 23081 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.063774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.0.98.62:40393)
0929 01:01:23.063912 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.063918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.065732 (+ 1814us) server_negotiation.cc:407] Connection header received
0929 01:01:23.065951 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.065955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.066008 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.066061 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.066722 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52990: BlockingRecv error: recv got EOF from 127.0.0.1:52990 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 01:01:23.068266 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.064555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.0.98.61:41127)
0929 01:01:23.064667 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.064671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.067395 (+ 2724us) server_negotiation.cc:407] Connection header received
0929 01:01:23.067567 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.067571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.067621 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.067670 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.068155 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33356: BlockingRecv error: recv got EOF from 127.0.0.1:33356 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 01:01:23.069590 23235 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.065160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.0.98.60:46489)
0929 01:01:23.065270 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.065274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.068715 (+ 3441us) server_negotiation.cc:407] Connection header received
0929 01:01:23.068884 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.068887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.068948 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.068998 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.069460 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52788: BlockingRecv error: recv got EOF from 127.0.0.1:52788 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 01:01:23.073940 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.072437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.0.98.61:41127)
0929 01:01:23.072536 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.072540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.073030 (+ 490us) server_negotiation.cc:407] Connection header received
0929 01:01:23.073238 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.073241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.073310 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.073357 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.073820 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33370: BlockingRecv error: recv got EOF from 127.0.0.1:33370 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 01:01:23.078764 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.077314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.0.98.61:41127)
0929 01:01:23.077409 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.077413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.077835 (+ 422us) server_negotiation.cc:407] Connection header received
0929 01:01:23.078019 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.078022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.078072 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.078120 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.078674 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33378: BlockingRecv error: recv got EOF from 127.0.0.1:33378 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 01:01:23.084190 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.082786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.0.98.61:41127)
0929 01:01:23.082878 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.082882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.083309 (+ 427us) server_negotiation.cc:407] Connection header received
0929 01:01:23.083486 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.083489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.083535 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.083583 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.084046 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33394: BlockingRecv error: recv got EOF from 127.0.0.1:33394 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 01:01:23.091821 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.090311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.0.98.61:41127)
0929 01:01:23.090443 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.090448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.090923 (+ 475us) server_negotiation.cc:407] Connection header received
0929 01:01:23.091116 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.091119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.091169 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.091218 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.091719 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33410: BlockingRecv error: recv got EOF from 127.0.0.1:33410 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 01:01:23.104223 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.102502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.0.98.61:41127)
0929 01:01:23.102628 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.102632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.103203 (+ 571us) server_negotiation.cc:407] Connection header received
0929 01:01:23.103412 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.103415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.103471 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.103525 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.104106 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33412: BlockingRecv error: recv got EOF from 127.0.0.1:33412 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 01:01:23.124786 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.122966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.98.61:41127)
0929 01:01:23.123085 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.123090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.123662 (+ 572us) server_negotiation.cc:407] Connection header received
0929 01:01:23.123886 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.123890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.123978 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.124041 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.124667 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33424: BlockingRecv error: recv got EOF from 127.0.0.1:33424 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 01:01:23.160882 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.159569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.0.98.61:41127)
0929 01:01:23.159741 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.159746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.159902 (+ 156us) server_negotiation.cc:407] Connection header received
0929 01:01:23.160098 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.160102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.160164 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.160265 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.160750 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33440: BlockingRecv error: recv got EOF from 127.0.0.1:33440 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 01:01:23.228354 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.227057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.0.98.61:41127)
0929 01:01:23.227236 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.227241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.227393 (+ 152us) server_negotiation.cc:407] Connection header received
0929 01:01:23.227602 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.227605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.227664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.227756 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.228213 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33444: BlockingRecv error: recv got EOF from 127.0.0.1:33444 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 01:01:23.359920 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.358709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.0.98.61:41127)
0929 01:01:23.358896 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.358900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.359010 (+ 110us) server_negotiation.cc:407] Connection header received
0929 01:01:23.359102 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.359105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.359157 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.359241 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.359758 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33454: BlockingRecv error: recv got EOF from 127.0.0.1:33454 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 01:01:23.619104 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:23.617846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.0.98.61:41127)
0929 01:01:23.617990 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.617995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.618096 (+ 101us) server_negotiation.cc:407] Connection header received
0929 01:01:23.618307 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.618310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.618375 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.618488 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.618950 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33464: BlockingRecv error: recv got EOF from 127.0.0.1:33464 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:01:23.919276 23262 catalog_manager.cc:1259] Loaded cluster ID: d2557d29b9b94f2984d245a6b2b6abdc
I20260929 01:01:23.919356 23262 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714: loading cluster ID for follower catalog manager: success
I20260929 01:01:23.921731 23262 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714: acquiring CA information for follower catalog manager: success
I20260929 01:01:23.923527 23262 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2e2479154ef42998e902e263d867714: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:01:23.970305 23289 catalog_manager.cc:1259] Loaded cluster ID: d2557d29b9b94f2984d245a6b2b6abdc
I20260929 01:01:23.970412 23289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9: loading cluster ID for follower catalog manager: success
I20260929 01:01:23.972041 23289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9: acquiring CA information for follower catalog manager: success
I20260929 01:01:23.973166 23289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9adb78cacc24913b4e65600fe4db7c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:01:24.137024 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.133042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.0.98.61:41127)
0929 01:01:24.133287 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:24.133291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:24.133404 (+ 113us) server_negotiation.cc:407] Connection header received
0929 01:01:24.133599 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:24.133602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:24.133735 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:24.133856 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:24.134116 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.134662 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.135534 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.135768 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.135880 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.136198 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:24.136624 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:24.136768 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:24.136805 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 01:01:24.136857 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260929 01:01:24.143287 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.139750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.0.98.60:46489)
0929 01:01:24.139975 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:24.139978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:24.139994 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:24.140075 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:24.140077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:24.140193 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:24.140299 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:24.140475 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.141012 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.141810 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.142058 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.142150 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.142485 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:24.142928 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:24.143050 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:24.143115 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 01:01:24.143168 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260929 01:01:24.150327 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.146747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.0.98.60:46489)
0929 01:01:24.146857 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:24.146863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:24.146877 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:01:24.147034 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:24.147037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:24.147081 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:24.147131 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:24.147504 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.148220 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.149125 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.149481 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.149593 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.149840 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:24.149926 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:24.150000 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:24.150105 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 01:01:24.150158 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:01:24.150527 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.146142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.0.98.61:41127)
0929 01:01:24.146282 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:24.146287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:24.146634 (+ 347us) server_negotiation.cc:407] Connection header received
0929 01:01:24.146833 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:24.146837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:24.146887 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:24.146942 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:24.147361 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.147895 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.149269 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.149613 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.149720 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.149987 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:24.150076 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:24.150144 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:24.150315 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 01:01:24.150372 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:01:24.151439 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.146211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.0.98.62:40393)
0929 01:01:24.146466 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:24.146472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:24.146490 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:01:24.146692 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:24.146696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:24.146853 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:24.146990 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:24.147171 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.148043 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.149692 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.150039 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.150139 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.150629 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:24.150984 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:24.151110 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:24.151214 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 01:01:24.151280 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260929 01:01:24.154492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 23010
I20260929 01:01:24.161231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 23085
I20260929 01:01:24.167161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIUjItr/build/debug/bin/kudu with pid 23160
2026-09-29T01:01:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3072 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6141 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285658 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST